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of Linux 


love my job. | teach Linux by day 


and write about Linux at night. 
It's easy to fall in love with your 
work when the things you do align 
with your passions. All of us here in 
the Linux Journal community have 

a love for Linux and open source, 
but even inside our world, there are 
some topics that are just downright 
fun! This month is full of articles 
we're passionate about. 

Joey Bernard starts off with a 
look at KStars. If you’re a space nut 
like me, you'll want to check out 
his detailed look at the KDE-native 
astronomy program. It’s dark enough 
to see the stars only at night, but 
with Joey's help, you can surf the 
night sky any time of day! Then, 
what better way to end a day of 
LCD star gazing than to read a 
tech book on your favorite subject? 


VIDEO: 
Shawn Powers runs 
through the latest issue. 
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For the Love 


SHAWN POWERS 


Reuven M. Lerner provides a 
breakdown of some of his top picks 
on topics from programming to 
podcasting. It’s hard to go wrong 
with a good book, and Reuven will 
help you find one. 

With the help of Dave Taylor, you 
will learn to deal with command 
arguments In scripts using getopt 
in your code. Dealing with 
arguments doesn’t seem like a big 
deal, but what if people combine 
them? (For instance, -rf instead 
of -r -f?) Rather than write pages 
of conditionals, Dave explores how 
to use getopt, which does all the 
dirty work for you. If you need to 
write a script that accepts command- 
line arguments, this month’s column 
is a must-read. 

| take a step away from the 
keyboard this issue and head around 
to the back of the computer— 
specifically, to the power cord. My 
family recently moved into an old 
turn-of-the-century house, and even 


with a massive electrical overhaul, the 
power is flaky at best. | decided to 
share my experience this month and 
clarify some of the various hardware 
options available to help regulate 

and stabilize the electricity coming 
into your home or business. The fact 
that I’m writing this on a desktop 
computer rather than a laptop means 
my hard work has paid off! 

Susan Sons has an incredible 
interview with Katerina Barone-Adesi 
this month where she gets the details 
on Snabb. If you've never heard of 
Snabb, you'll be glad you read the 
interview, because it’s an incredible 
network toolkit that isn’t built in 
to the Linux kernel. If the idea of 
bypassing the Linux kernel to increase 
speed seems odd, you definitely want 
to read this interview! 

You might remember Kyle Rankin 
writing a few years ago about using 
a Pogo Linux device (and later a 
Raspberry Pi, | think) to keep his 
homebrew beer chilled while it 
aged. Klaus Kolle takes a different 
approach, and using a BeagleBone 
Black, he controls the heating of 
the mash during his beer-making 
process. Whether you need to keep 
your homebrew hot or cold, it turns 
out Linux can be the answer in either 
case! Check out his cool project this 
month, and potentially improve your 
beer while improving your geek cred. 


CURRENT_ISSUE.TAR.GZ 


Sol Lederman finishes off the 
issue with some awesome problem 
solving. The best use of Linux, or 
technology in general, is to make 
quick work of something that would 
take a long time to do on your own. 
Scripting was designed for that 
exact purpose. In this command-line 
tutorial, Sol describes how to write a 
one-liner to find a specific date, and 
then put that one-liner into a script 
to make it more flexible. Most good 
programs start with a problem that 
needs to be solved, and in this case, 
the problem is trying to find the next 
Friday the 13th. 

We all have passions that drive us 
to learn and explore. For some of 
us (me!), it’s feeding the birds. For 
others, it’s brewing the perfect beer. 
The best part about being a Linux 
geek is that we can take advantage 
of Linux’s open-source nature and 
make our passions really come to 
life. We hope you share some of 
the excitement we brought into this 
issue and can’t wait to hear about 
what sorts of projects you improve 
with the help of our favorite OS!m 


Shawn Powers is the Associate Editor for Linux Journal. 
He’s also the Gadget Guy for LinuxJournal.com, and he has 
an interesting collection of vintage Garfield coffee mugs. 
Don't let his silly hairdo fool you, he’s a pretty ordinary guy 
and can be reached via e-mail at shawn @linuxjournal.com. 
Or, swing by the #linuxjournal IRC channel on Freenode.net. 
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letters 


Server Hardening 
ab Regarding Greg Bledsoe’s “Server 
Encryption Hardening” article in the November 
2015 issue: great article—lots of 


File Transfer shane detailed help in one source. One 
Security a question: is it possible to mention 
Ys Making specific logs you reference in the 
Me G9 Oi Sins article? | get lost quickly when seeing 
[Zak the large number of logs scattered 
TD ees about my Debian server. 
habiee —Tom Browder 
AuditSefid O ke 
Console Access ” v Bug in Script 
| believe that the final version of the 
Hardware Interrogation script that Dave Taylor came up with 
I've just read Federico Kereki's in his Work the Shell column titled 
article about interrogating a “Analyzing Comma-Separated Values 
Linux system titled “What's in the (CSV) Files” in the December 2015 issue 
Box? Interrogate Your Hardware” of Linux Journal contains an oversight. 
in the December 2015 issue. | Specifically, it does not handle the case 
love this kind of article and hope in which more than one field contains 
to see more! commas (for example, the dollar 
—Brian Clark amount field and the comment field). 
| have modified Dave's script to take 
Federico Kereki replies: Thanks, this into account. Hopefully, this will 
Mr Clark, for your kind words. The be of some help. | always enjoy Dave's 
article grew out of my actual need column and have learned a lot from it. 
to know about the hardware in Here’s the modified script: 
my own machine, and because of 
Linux’s openness, | learned even #! /bin/bash - 
more than | had expected. I’m glad 
you liked my results! # fixesv 
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# fix CSV files with embedded commas 


# The problem is that some spreadsheet fields may contain 
commas. In the sample case, this includes the dollar 
amount and comment fields. I believe you overlooked the 
case in which both the dollar amount and comment fields 
contain commas. Your script assumes that there is at 


most one such instance. 


# The simplest solution is to export the spreadsheet 
contents with some field delimiter that can never appear 
in any field, e.g., a tab. Then write the script 


using this delimiter. 


# Original code 


# while read inline 

# do 

# if [ ! -z "$(echo $inline | grep \")" J 
# then 

# fi=$(echo $inline | cut -d\" -f1) 


# f2=$(echo $inline | cut -d\" -f2) 


# f3=$(echo $inline | cut -d\" -f3) 
# echo $filecho $f2|sed 's/,//g' “$f3 
# else 

# echo $inline 

# fi 

# done 

# exit 0 


# This works correctly ONLY when there is EXACTLY ONE 


field enclosed in double quotes. 


# Revised code 


| LETTERS | 


# For each line that contains at least one field enclosed 
in double quotes, process each such field from left to 
right until no fields are enclosed in double quotes and 
all remaining commas are field separators. The steps are: 
(1) replace the double quotes enclosing the field being 
processed with a temporary delimiter to isolate that 
specific field, (2) remove any commas embedded in the 
isolated field, (3) reconstruct the line without the 
temporary delimiters. The temporary delimiter must be 

a single character (for the cut command) that cannot 
appear in the input file. I selected an asterisk (*), 

but other characters can be used. Some characters (such 
as asterisk, colon, hyphen, and equals) work fine, while 


others (such as tab and semicolon) do not. 
td=* # temporary delimiter 


while read inline 
do 
while [ ! -z "$(echo $inline | grep \")" ] 
do 
inline=$(echo $inline | sed "s/\"/$td/" | sed "s/\"/$td/") 
f1=$(echo $inline | cut -d"$td" -f1) 
f2=$(echo $inline | cut -d"$td" -f2) 
f3=$(echo $inline | cut -d"$td" -f3) 
inline=$(echo "$fi$(echo $f2 | sed 's/,//g')$f3") 
done 
echo $inline 
done 


exit 0 
# Test input file fixcsvtest.txt: 


$ cat fixcsvtest.txt 
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4/7/14, subscriptions, 199.99,Ask Dave Taylor Monthly 
4/10/14, subscriptions, "1,300.99",Linux Journal 
4/10/14, subscriptions, "1,300.99","Linux Journal, APR 2014" 


4/10/14, subscriptions,19.99,"Linux Journal, annual" 


ab,cd,ef,gh 
ab,cd,ef,"g,h" 
ab,cd,"e,f",gh 
ab,cd,"e,f","g,h" 
ab,"c,d",ef,gh 
ab,'"¢,d", ef, "g,h" 
ab;"67d""6,F"gh 
ab;'"e, 0", “Ne, to, h" 
"a,b",cd,ef,gh 
"a,b",cd,ef,"g,h" 
"a,b",ed,"e,f",gh 
“a, b",ed,"e,T", "gh" 
"a,b","¢,d",ef, gh 
"a,b", "¢e,0" ef, "gi h" 
"ab "epd", “ey Fgh 
"a,b" ,"¢,d""e,F","g, h" 


$ 
# Test Results 


$ ./fixcsv < fixcsvtest.txt 

4/7/14, subscriptions, 199.99,Ask Dave Taylor Monthly 
4/10/14, subscriptions, 1300.99,Linux Journal 

4/10/14, subscriptions, 1300.99,Linux Journal APR 2014 


4/10/14, subscriptions,19.99,Linux Journal annual 


ab,cd,ef,gh 
ab,cd,ef,gh 
ab, cd,ef,gh 


ab,cd,ef,gh 
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ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 
ab,cd,ef,gh 


$ 
—Jeff Mumma 


Dave Taylor replies: Thanks for your 
note, Jeff, and | do believe you're 
correct that | didn’t test the case 
where more than a single field of the 
input data included commas. Bah, 
pesky debugging! | like your mods, 
and yet still have a niggling sense that 
the entire problem can be sidestepped 
with the perfect regular expression. If 
! only had a few weeks to create it! 


Photo of the Month 

| thought you would like to see an 
unusual place where LJ is being read 
this month: 49 degrees north, 35 
degrees west. That’s the middle of the 
Atlantic Ocean at 20 knots heading 
for NYC. The satellite Internet costs 
are rather steep on board so | brought 


LETTERS 


a few issues with me. Must dash as the sun is 


over the yard arm. At Your Service 


Dest InNauONn 
New York 
SA 


* 427N 


) fr 
0) 


atic. 


CONTROL 
Ou! 


3D PRINTER 


SECURITY || rn real-world stories forthe magazine. 


—Roger Greenwood 


Remember, send your Linux-related VERTISING: Linux 
photos to Ijeditor@linuxjournal.com! Be ee ek 


UU TEURRTOPURTOOURROCORROUORORUUHERUURROUDRROUDRNORDRTOUDIIOURTIOOOEL 
WRITE LJ A LETTER 


We love hearing from our readers. Please Pee ee es 
send us your comments and feedback via 
http://www.linuxjournal.com/contact. 
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UP 


diff -u 


What’s New in Kernel Development 


The OOM killer is a tough nut to 
crack. How can a system recover 
when it’s violently thrashing and out 
of RAM? Once upon a time, you'd 
just have to reboot. And today, that 
still might be necessary, but less so, 
because the OOM killer attempts to 
identify and stop the process that 
seems to be causing the hangup. 
The problem is, it may not choose 
the right process every time. Another 
problem is that the whole thing Is 
super tough and complex. 

Michal Hocko recently tried to 
peel off a sliver to work on, taking 
the lead from Mel Gorman and 
Oleg Nesterov. Apparently, the 
current OOM killer would allocate an 
extra batch of memory just for the 
process it wanted to kill to actually 
give it enough breathing room to 
terminate properly. But under some 
circumstances, the process would 
accept the extra memory and still 
hang the system. Then with no more 
memory to dole out, the OOM killed 
couldn't try again, and it was time to 
hit the reset button. 
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Michal posted a patch to create a 
new kernel thread that would reclaim 
that extra memory if it went unused. 
Then the OOM killer could try the 
same thing on a different process 
and hopefully have a different result. 
And although there were no major 
objections to Michal’s patch itself, 

a variety of folks objected to the 
idea of making any kind of 
incremental improvement to the 
OOM killer, when the Big Problem 
had not yet been solved. 

The Big Problem, as described 
by Johannes Weiner, was how to 
resolve memory deadlocks in general. 
Only by solving that problem could 
the OOM killer successfully kill the 
processes it needed to, even to the 
point of killing all user processes, just 
to keep the kernel up. 

But, Michal made a point of keeping 
the discussion clamped down to a 
consideration of only the small fixes 
he'd proposed. He acknowledged 
that he had no solution for the Big 
Problem, and he pointed out that 
no one else seemed to have a viable 


solution for the Big Problem either. 
And until something viable came up, 
Michal saw no point in stalling OOM 
killer development. If something could 
be done to improve it, he felt, then it 
should be done. 

By and large everyone went along 
with this, but still, it’s clear there's 
a lot of pressure on the OOM killer 
system to come up with some kind of 
new idea or at least to create a policy- 
based system that puts control of the 
choices of processes to kill into the 
hands of system administrators rather 


than the kernel algorithms themselves. 


Linus Torvalds had some advice 
for anyone writing kernel code 
that needs to lock resources: it’s 
probably better to use existing locking 
implementations rather than rolling 
your own—at least, until you know 
what you're doing. As he put it: 


People need to realize that locking 
is harder than they think, and not 
cook up their own lock primitives 
using things like trylock without 
really thinking about it a /ot. 


Basically, trylock() on its own 
should never be used in a loop. 
The main use for trylock should 
be one of: 


1) Thing that you can just not do at 


(| UPFRONT | 


all if you can’t get the lock. 


2) Avoiding ABBA deadlocks: if 

you have an A->B locking order, 
but you already hold B, instead of 
“drop B, then take A and B in the 
right order”, you may decide first 
to trylock(A), and if that fails, 
you then fall back on the “drop and 
relock in the right order”. 


But if what you want to create is 

a “get lock using trylock”, you 
need to be very aware of the cache 
coherency traffic issue at least. 


It is possible that we should think 
about trying to introduce a new 
primitive for that loop_try_lock() 
thing. But it's probably not common 
enough to be worth it—we've had 
this issue before, but | think it’s a 
“once every couple of years” kind of 
thing rather than anything that we 
need to worry about. 


The “locking is hard” issue is very 
real, though. We've traditionally had 
a /ot of code that tried to do its own 
locking, and not getting the memory 
ordering right, etc. Things that 
happen to work on x86 but don’t on 
other architectures, etc. 


—ZACK BROWN 
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Android Candy: 


Quick Games 


The biggest problem | have with 
gaming is that it takes far too long to 
get “into” games. I’m generally very 
busy, and my gaming time usually lasts 
as long as it takes for the dentist to 
call me in from the waiting room (or 
possibly how long it takes me to use 
the bathroom, but e/w, let’s not go 
there). For me, the perfect game can 
be fun even if | can play only for a few 
minutes. It also has to be very quick to 
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learn, because “learning to have fun” 
isn’t very much fun at all. 

There are a few old standbys that 
work well: Bejeweled, Angry Birds, 
Peggle, Plants vs. Zombies, Candy 
Crush and so on. The problem is, 
although those games are fun, | grow 
tired of them fairly quickly—either that 
or they're so addictive | fear my family 
will disown me for ignoring them 
while | play just one more level. 


Here are some games I've been playing lately: 


™ Swish: it's a puzzle game, but it’s just different 
enough that | find it fun. The physics seem 
“right” to me while playing, and the graphics 
are really great. The premise is that you're an 
alien playing basketball in space. You know, like 
aliens always do. There are 60 levels, and there's 
enough of a challenge to make it fun. 


m Dumb Ways to Die 2: | loved the first Dumb Ways 
to Die game. It was absurd. It was funny. It was 
cartoonishly morbid. Part 2 is more of the same, 
and that’s a very, very good thing. 


m Asphalt Series: there are a bunch of games in 
the Asphalt racing series. Some of them work 
better than others, and all of them are fairly 
large downloads. | like Aspha/t because you 
get to race really cool cars really fast. If you 
ever played Burnout on the PlayStation, Asphalt 
games will seem pleasantly familiar. You have 
to pay for gas if you play for too long, but | 
generally don’t have enough time and play each 
session only until my gas Is gone. 


What are your favorite “five-minute games” 
for the Android platform? | never can really get 
into a book in that short amount of time, so it 
tends to be the only time | play games. If you 
have any suggestions, drop me a message at 
shawn@linuxjournal.com and I'll try to follow up 
in later months with the best of the best. 

Note: you can find all the games mentioned here 
in the Google Play Store.-SHAWN POWERS 
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They 
Said It 


Sometimes we 
do athing in 
order to find 

out the reason 
for it. Sometimes 
our actions 

are questions 
not answers. 
—John Le Carré 


The greatest 
justice in life 

is that your 
vision and 
looks tend to go 
simultaneously. 
—Kevin Bacon 


There are some 
things you learn 
best in calm, and 
some in storm. 
—wWilla Cather 


The only true 
happiness comes 
from squandering 
ourselves for 

a purpose. 
—William Cowper 


If your ship 
doesn’t come in, 
swim out to it! 
—Jonathan Winters 
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Non-Linux FOSS: Snk 


Praise for Snk: 


“It's lovely, dear” —Mowdgliis mom 
“Good job, bro.” —Mowdgliis sister 
“Yeeeaaaahhh!" —Mowalii 
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I'm apparently in a silly-game mood 
this month, because | stumbled 
across an open-source project | 
couldn't keep all to myself: Snk. If 
you remember the classic game of 
snake, Snk is the same concept, but 
smaller, harder and with music. 

| actually really like the Snk 
program because it’s fairly simple, 
and the developer (Mowglii) 
has put the Xcode project on GitHub 
for folks to download. 
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If you're just starting with OS X 
Swift development, Snk is a project 
you can tweak for some learning 
on the fly. 

My favorite part of the project is 
ace ue “reviews” for It. edd 
over to http://www.mowglii.com/snk 
to grab a and its source eae 
today. And, good luck with level 
three; | run directly into the wall 
every time! 

—SHAWN POWERS 
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Handheld Emulation: 
Achievement Unlocked! 


| love video game emulation. My own the original cartridges, so I’m 
favorite games were produced in not going to tell you where to find 
the 1980s and 1990s, so if | want ROMs to download or anything like 
to play them, | almost always have that. What | am going to share is 

to emulate the old systems. There my recent discovery of the perfect 
is usually a legal concern about handheld gaming system. Oddly 
ROM files for games, even if you enough, it was never intended to be 


ACI eior 


(Image from http://wololo.net) 
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an emulator. 

The PSP is truly incredible 
hardware. The PSP Vita is its 
bigger, younger sibling, but if 
you have an old PSP, | urge you 
not to throw it away. With a 
simple firmware hack (also legally 
questionable, | suppose), it’s 
possible to load emulators that 
will play Atari, NES, SNES, Game 
Boy, Genesis, PS1 and most other 
console games almost flawlessly. 

| never had a PSP, but | was 
able to get a PSP Go in mint 
condition on eBay for $89. The 
PSP Go comes with 16GB of 
storage, so you don’t even need 
to get its proprietary memory 
card to load it up with games! 

One of the problems with the 
emulation scene is that sites seem 
to come and go fairly regularly. | 
found all the information | needed 
to get my PSP Go ready to play 
Mario by doing some Google 
searching for PSP emulators. 
Specifically, this page was great: 
http://wololo.net/emulators-for-the- 
psp-ps-vita-the-ultimate-download-list. 

If you already have a PSP device, 
the instructions for custom firmware 
installation is simple. If you don’t 
have one, deciding which version 
of the PSP to purchase is one of the 
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toughest steps. If you like the larger 
layout, | recommend the PSP 2000 
model. It has an incredible screen 
and fewer buggy design choices 
than the original. If you're looking 
for portability, I’m very fond of the 
PSP Go | purchased from eBay. The 
screen is smaller, but it’s still plenty 
large and has beautiful quality. 
Good luck, and have fun! 

—SHAWN POWERS 
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Astronomy for KDE 


Although | have covered a large this article, I’m starting off with the 
number of science applications in this KStars astronomy program. 

space, | haven’t really looked at too If you have the full KDE environment 
many options available within the KDE installed, you already should have it 
desktop environment. This has been available. If you don’t, you should 

due to my own biases in using a GTK- be able to install it. For example, you 
based desktop environment, but now can Install KStars on Debian-based 


I'd like to look at some of the packages distributions with this command: 
available for people who really like to 
use KDE on their own machines. So in sudo apt-get install kstars 


e Setup Wizard — KStars 


Welcome to the KStars Setup Wizard 


seeee, 'his wizard will help you set up some basic options, such as your location on 
: S Earth. 
, To get started, press the Next button. 


Next > Cancel 


Figure 1. When you first start KStars, you need to go through the setup wizard. 
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If this is the first KDE-based 
application that you are installing, 
it also will need to pull in a rather 
large set of dependencies—that'’s 
just the price of using a new 
GUI toolkit. Of course, any other 
packages will be incrementally 
smaller since all of the shared 
dependencies already will be there. 

To launch KStars, you either can 
click on a menu item in your desktop 
environment or enter the kstars 


© KStars 


File Time Pointing View Tools Data Observation Settings Help 


Zoomin Zoom Out Find Object 


LT: 4:17:19 PM AST Sunday, December 6, 2015 


Fredericton, New Bru 


Empty sky 
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command in a terminal window. 

The first time you start KStars, 

you need to go through the setup 
wizard to configure elements like 
your location. Once KStars finishes 
starting up, you should see a display 
of the sky from the location you set 
during the setup. 

You can pan the display around 
simply by clicking and dragging the 
star field to see locations of interest. 
The items that are labeled depend 


Advance one step backward in time Stop Clock Advance one step forward intime 1 sec 


nothing 


+197° 37' 35", +36° 43' 12" 19h 55m 18s, -5° 50' 50" 


Figure 2. You get a display of the sky from your location when KStars starts. 
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e KStars 


File Time Pointing View Tools Data Observation Settings Help 
Zoomin Zoom Out Find Object 


LT: 8:52:44 PM AST Sunday, December 6, 2015 


Fredericton, New Brunswick, Canada 


+LZ 


Advance one step backward in time Stop Clock Advance one step forward in time | 1 sec 8) eae 


NGC 6793 
Open Cluster 
Vulpecula 
Rise time: 7:07:48 AM AST 
Set time: 10:32:24 PM AST 
Transit time: 2:49:07 PM AST 


Center & Track 
Add flag... 


Angular Distance To... 


Starhop from here to... 


Details 

Attach Label 

Add to Observing WishList 
Show SDSS Image 

Show DSS Image 


+286° 29' 07", +15° 01' 56" 19h 23m 34s, +22° 06' 11" 


Figure 3. You can interact with the objects in the display. 


on your zoom level. Two buttons at 
the top of the display allow you to 
zoom in and out. Beside these is a 
third button, labeled Find Object, 
that you can click on to get a search 
window. This window can use filters 
to search for particular types of 
objects, like comets or asteroids, or 
you can search through all of the 
objects that KStars knows about. 
Once you find an object of 
interest, you can click on it and have 


24 / FEBRUARY 2016 / WWW.LINUXJOURNAL.COM 


the display move around until the 
object is in the center of the display. 
If it is below the horizon, a warning 
box will pop up asking if you still 
want to re-center the display. 
Once you have selected your object, 
you can right-click on it to get a 
drop-down menu of things you can 
do with it. 

The header of this drop-down 
contains the full name of the object, 
along with rising and setting times. 
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e Object Details — KStars 


General Position Links Advanced 


A7 star in Aquila 


Distance: 


Log 
Altair (a Aquilae), HD 187642 


Magnitude: 6.93 mag 


4.94 pc 


B - V index: -- 


Add to Observing List 


Center in Map 


Center in Telescope 


Close 


Figure 4. The detail window has a lot of extra information available for most objects. 


Below that, the first option is to 
center and track the object. This is 
useful because the default display 
mode Is to have the display updated 
in real time. This way, you always 
have a view of what the sky looks 
like right now. You can select the 
details option to pull up even more 
information on the object. 

You can add a flag to make the 
object easier to keep track of or 
even add tracks as the object moves 


across the field of view. You also 
can calculate the angular distance 
to another object or plan out a star- 
hopping path to some other object. 

For some objects, there may be 
images from the Deep Sky Survey 
available. If they are, there will be 
options on this drop-down menu 
labeled “Show DSS Image” or “Show 
SDSS Image". 

Although KStars comes with quite 
a bit of data when you install it, this 
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e Get Hot New Stuff — KStars 


KStars Add-On Installer 


Steinicke NGC/IC Catalog 


By Jason Harris jharris@30doradus.org 
More complete NGC/IC catalog 


Abell Planetary Nebulae Catalog 


By Carl Knight SleeplessAtKnight@gqmail.com 
Catalog of Abell Planetary Nebulae 


Sharpless HII region Catalog 


By jasem Mutlaq mutlagja@ikarustech.com 
The Sharpless (Sh2) catalog of HII regions (diffuse 


nebulae) 


7 = 


install 
Details 
Order by: 
© Newest 
Rating 
Install Most downloads 
Details 
Installed 
Search: 
iter search phi 
install 
Details 
Close 


Figure 5. Several extra data sets are available that you can download and install. 


isn’t everything that is available. You 
can add new data sources by clicking 
on the menu item Data—Download 
New Data. This will open up a 

new dialog window giving you a list 
of what data catalogs are available 
to download. 

For some of the data sets, like 
comet and asteroid information, 
there is a constant updating of 
the detailed information available 
to the astronomical community. 

You can download those updates 
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by clicking on the menu item 
Data—Updates— and selecting 
which catalog to update. You 
even can import your own data by 
going to the configuration window 
and clicking the Import Catalog... 
button on the Catalogs section. 
When you are in the configuration 
window, you can see that you also 
can change settings on how to 
view several different categories, 
such as solar system objects, 
satellites and supernovae. 


e Telescope Wizard — KStars 


Help Cancel 


Please click next to continue. 


(| UPFRONT | 


Welcome to the KStars Telescope Setup Wizard! 


This Wizard will help you to connect to your telescope and control it from 
KStars. You will be asked to verify some basic information. 


You can get extended information on telescope support in KStars by pressing 
the help button at any point during the Wizard. 


Next 


Figure 6. You can add a telescope as a device under the control of KStars. 


The guides section defines what 
extra information is displayed, such 
as constellation lines, names and 
the Milky Way. The INDI (Instrument 
Neutral Distributed Interface) section 
controls how KStars talks to your 
connected telescope. You can pull up 
the telescope wizard by clicking on 
Tools—Devices— Telescope Wizard. 

Once the connection Is made, 
you can pull up the control panel 
and send instructions to your 
telescope. You can even automate 
your observations using this 
functionality. Clicking on the menu 


item Observation—Observation 
Planner will pop up a window where 
you can define a complete set of 
observations you want to run. If it is 
a bit confusing at first, a wizard is 
available to help walk you through 
setting up a plan. 

While you will make your own 
observations, you also may want 
to look at observations made by 
other researchers. The file format 
most often used is the FITS format 
(Flexible Image Transport System). 
KStars includes a FITS viewer, which 
has a number of analysis tools. 
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e INDI Control Panel— KStars 


LX200 Basic 


Main Control Options 


Connection Connect Disconnect 
© Ports Port /dev/ttyUSBO /dev/ttyUSBO 


Object - ‘ 


0:00:00 0:00:00 
Equatorial JNow 
Dec D:M:S 0:00:00 0:00:00 
On Set Slew Track Sync 
Abort .../Track Abort 
4 > 


| 2015-12-07T01:42:07: Error connecting to port /dev/ttySO. Make sure you have BOTH read and write 
permission to the port. 


Clear Close 


Figure 7. Once it is connected to your computer, you can use KStars to control your telescope. 


You can open a FITS file by clicking with the data in the image. You can 
on File—Open FITS. look at the basic statistics of the image, 
Several tools are available to work including the width, height, maximum, 
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e KStars FITS Viewer — KStars 
File Edit View Help 
Open Save Save As Redo Zoomin ZoomOut Actual Size Rotate Right Rotate Left Flip Horizontal Flip Vertical Histogram Statistics Auto stretch 


UlTfuv2582gc.fits X 


12h 09m 46s, 39° 30' 02" 0.00 X:386 Y:357 100% 512x512 *) 


Figure 8. The FITS viewer lets you analyze and manipulate observational images. 


minimum and mean of the pixels. You 
can pull up a histogram, showing the 
frequency spread of the image data. 
The View menu item includes several 
more tools where you can equalize the 
image, pass It through a high contrast 
filter or apply an auto stretch. There is 
even an entry to mark any stars that 
KStars can identify automatically. 

You now have another astronomy 
program in your toolkit that you can use 
when your scientific research moves in 
that direction. In the coming months, | 
plan to look at other scientific software 
packages within this desktop environment 
and see just how much research can be 
done with KDE.—JOEY BERNARD 
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Poppins 


My friend and fellow Linux Journalian 
Kris Occhipinti recently posted a 
reminder on Facebook for everyone to 
back up regularly in 2016. Although 
it’s something we already should be 
doing, if you’re not a regular backer- 
upper, you should start today! The 
method of backup isn’t nearly as 
important as the act itself, but this 
month, | found a new project that 
simplifies the backup process nicely. 
Poppins is an open-source project 
that builds on the SSH and rsync 
programs to create an incremental 
backup system that is simple, fast 
and reliable. Tons of other backup 
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programs are 
available, but 
Poppins doesn’t 
try to be a full-blown system; rather, 
it’s a simple one-liner that will do file 
rotation, snapshots and more. It can 
be automated with cron, or you can 
run it manually from the command 
line. (But you should really, really 
make a cron job!) 

Other backup systems have more 
robust interfaces, restoration options 
and so on, but the beauty of Poppins 
is its simplicity. It’s just one step above 
manually running rsync yourself, but 
that one step means it’s simpler to 
do. In my world, simple is about the 
only way to make sure something gets 
done at all! 

In fact, even though it’s a new 
project and still in beta, Poppins gets 
this month's Editors’ Choice award. 
It’s simple enough that you might 
actually get around to using it, and it 
has enough features to really benefit 
you in the case of a catastrophic 
failure. Check it out today at 
http://poppinsbackups.wordpress.com, 
or go right to the Bitbucket page: 
http://bitbucket.org/poppins. 

—SHAWN POWERS 
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In the computer industry, you 
don’t have a choice—you must 
constantly keep learning new things, 
just to stay In place. | soend many 
hours each week reading blogs, 
watching conference videos and just 
exploring technologies on my own, 
because if | don’t do that, I’m going 
to find myself unable to use the latest 
technologies to help my clients. 

But of course, there is at least one 
other tried-and-true way to learn: 
books. Sure, books (or their electronic 
equivalents) have been eulogized 
numerous times during the last few 
years. But, programming-related 
books continue to be published, 
some by well known publishers and 
others by independent programmers. 

Every so often, | take time in this 
column to review the latest crop of 
books that have helped make me 
more effective in my work. As always, 
the descriptions and reviews of these 
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books (and other media) are not only 
subjective in terms of the content, 
but also in terms of the timing; it’s 
quite possible that | only recently 
discovered a book that came out a 
while ago. | try to mention books 
here that you might find interesting 
or useful in your work, much as I've 
found them to be interesting and 
useful in my own work. 

Perhaps the most interesting 
trend we're now seeing in the 
book market, and especially when 
it comes to computer books, Is the 
number of self-published ebooks 
being produced. (Heck, I’ve written 
two of them myself: one on Python 
and another on regular expressions.) 
These books rely on word of mouth, 
social media and marketing by the 
developers/authors themselves. 

There’s no doubt that traditional 
publishers continue to have great 
reach and are producing more books 


COLUMNS 


AT THE FORGE 


But, if you’re trying to teach children to program 
and are agnostic on the language used, I’m not sure 
if Ruby is the right way to go. 


(and well edited books, at that) than 
independent developers ever can 
hope to match. But the number of 
self-published ebooks continues to 
grow, as well as those sold via sites 
like https://leanpub.com. It will be 
interesting to see what happens in 
the coming years as book-creation 
systems, such as GitBook and 
Softcover, make it even easier to 
create and sell your own books. 

So, what have | been reading, and 
what would | suggest you read as well? 


Programming Books 
| continue to enjoy reading 
programming books, partly because 
there’s always something new to 
learn, partly because different 
authors offer different perspectives, 
and partly because | often can 
pick up insights and tricks that | 
can incorporate into my courses. | 
continue to use Ruby on Rails for 
Web development, so | always am 
happy to look at Ruby-related books. 
One of my most favorite Ruby books 
is Metaprogramming Ruby by Paolo 


Perotta and published by the Pragmatic 
Bookshelf. The book has been updated 
to include Ruby 2, and although 
Ruby’s underlying object system hasn't 
changed in many years, every version 
of Ruby is a bit different—and knowing 
how to take advantage of those 
differences can make a huge difference 
in your productivity. If you are using 
Ruby in your work, you really should 
read this book. 

Teaching children to program 
always has been a passion of mine, 
so I'm always interested in books 
and software that try to teach kids 
how to program. Several years ago, 
| mentioned the book Python for 
Kids, and it shouldn't be much of a 
surprise to discover that there’s now 
a book for kids to get into Ruby, 
called Ruby Wizardry. | think | should 
distinguish between two different 
goals here though. If you really want 
your children to learn Ruby, this book 
is an entertaining way to do so, with 
numerous examples and a storyline 
I'd like to think kids would enjoy. 
But, if you’re trying to teach children 
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to program and are agnostic on the 
language used, I’m not sure if Ruby is 
the right way to go. (See below for my 
recommendation of an on-line system 
that seems to be doing that right.) My 
limited experience with trying to teach 
Ruby, or even Python, to children has 
been fairly lackluster—not because the 
materials are bad, but because kids are 
less interested in the language than in 
what they can do with the language. 

| have been using a great deal of 
Python in recent years and also have 
been teaching a huge number of 
Python classes around the world. I’m 
always looking for ways to explain 
advanced aspects of Python better; 
thus, Obi Ike-Nwosu’s self-published 
Intermediate Python ebook caught 
my attention. | found that many 
aspects of this book put ideas into 
nice context and into a focus that 
otherwise might have been lost. If 
you have been working with Python 
for more than a few months and 
want to deepen your knowledge and 


understanding on a few topics, this 
book seems like a good place to go. 

Finally, | continue to work a great 
deal with PostgreSQL, a database 
whose features and community 
continue to grow in size and quality 
with every passing year. Hans- 
Jurgen Schonig, a very accomplished 
PostgreSQL consultant and contributor, 
has written Troubleshooting 
PostgreSQL (published by Packt), a 
book that is something like a first- 
aid kit for PostgreSQL developers 
and DBAs. He touches on all of the 
sensitive topics, including indexing, 
transactions, locking, monitoring, 
replication and backups. Even if you 
have been using PostgreSQL for some 
time, Troubleshooting PostgreSQL is 
an excellent investment, one that will 
more than pay for itself in time saved 
and hair (not) pulled. 


Data Science 
It was probably somewhat inevitable 
as someone who has worked with 


Even if you have been using PostgreSQL for some 
time, Troubleshooting PostgreSQL is an excellent 
investment, one that will more than pay for itself 
in time saved and hair (not) pulled. 
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databases, uses Python and has an 
interest In statistics that | would 
slowly but surely slide into data 
science. I’ve thus been spending time 
during the last year, and especially the 
last few months, learning more and 
more about this fascinating subject— 
both so | can help my clients solve 
data-science problems and so | can 
offer training in the use of Python 
for data science. There has been an 
explosion of books about data science 
in the past few years, and I’ve been 
working my way through no small 
number of them. 

Two high-quality books about 
data science are Doing Data Science 
by Cathy O'Neil and Rachel Schutt, 
published by O'Reilly, and Data 
Science from Scratch by Joel Grus, 
also published by O'Reilly. Grus 
uses Python and pushes hard in 
the use of functional programming 
techniques in Python, along with the 
standard data-science stack available 
to Python developers, such as 
matplotlib and pandas. But, he also 
provides no small amount of insights 
into how—and, just as important, 
when and why—we should use such 
techniques as linear regression vs. 
multiple regression vs. k-nearest 
neighbors. The book contains a very 
large number of code examples, 
and because it’s written in Python, 


it’s likely to use paradigms with 
which Python developers already 
are familiar, such as iteration and 
comprehensions. The former book, 
which | have recommended in 
earlier book-review columns, gives 
a serious introduction to the how 
and why, not only including code, 
but also the mathematical ideas 
behind that code. 

Another book on this topic, 
currently in early release from O'Reilly, 
is Python Data Science Handbook 
written by Jake VanderPlas. If Doing 
Data Science takes a mathematical 
approach, and Data Science 
from Scratch takes a functional- 
programming approach, then this 
book takes what I'd call a practical 
approach—introducing the elements 
of Python that a budding data 
scientist most needs to understand, 
keeping the programming and 
mathematics at a reasonable level for 
people who are unfamiliar with either 
or both. I’m still reading through 
this early-release edition, but as an 
introduction to learning science with 
a relatively easy learning curve, it may 
soon become my most-recommended 
book for people new to the subject. 

lvan Idris has written two books 
for Packt Press that are good overall 
introductions to data-science tools in 
Python, namely Python Data Analysis 


WWW.LINUXJOURNAL.COM / FEBRUARY 2016 / 35 


AT THE FORGE 


AT THE FORGE 


COLUMNS 


and Python Data Analysis Cookbook. 
The former introduces the topics and 
(as with the book by Grus, mentioned 
previously) provides a tutorial into 
areas like visualization, time series and 
machine learning. The cookbook is 

a complementary volume; instead of 
teaching based on subjects, it takes a 


problem-solution approach. I’d suggest 


that reading the tutorial first is a good 
idea; you then can read through the 
cookbook’s table of contents and/or 
index to find the specific issues on 
which you need to work. 


If you're interested in data science, 


but would rather hear about the 
insights and perspective than the 
mathematics, read Dataclysm: Who 
We Are (When We Think No One Is 
Looking) by Christian Rudder, one 
of the founders of OKCupid. In 
this book, Rudder finds fascinating 
correlations that provide insights 
into human behavior—thanks to 
the huge number of data points 
collected by the dating site he runs. 
| would be remiss if | were to 
ignore the amazing e-mail list 
produced by Analytics Vidhya, a name 
that doesn’t quite roll off of the 
tongue of English-speaking Americans, 
but which offers daily insights for 
anyone interested in the technology, 
algorithms and practice of data 
science: http://analyticsvidhya.com. 


36 / FEBRUARY 2016 / WWW.LINUXJOURNAL.COM 


Other Subjects 
| might be a software developer, but 
| also run a small company, so I'm 
always looking for books that can 
help me run my company even better 
or find new business opportunities. 
One book, published just before 
| wrote this column, comes from 
a group of consultants with a 
Slack channel. They jointly wrote 
and published The independent 
consulting manual, an ebook meant 
for people who are consulting, or 
want to consult, but aren’t sure 
where to start, how to find clients, 
how to price themselves or how to 
create products. The book (at http:// 
independentconsultingmanual.com) 
has many famous contributors, all of 
whom have made the leap to running 
their own businesses, and the lessons 
they've learned, and contributed, make 
for extremely worthwhile reading. 
One of the authors of the above 
ebook, and a co-panelist of mine 
on the Freelancers Show podcast, 
is Philip Morgan. Morgan has done 
a lot to encourage consultants to 
position themselves—that Is, to 
define a specific niche in which they 
can be the best-known experts. His 
book, The Positioning Manual for 
Technical Firms (available at 
http://ThePositioningManual.com) is 
full of excellent advice, including 
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In What If, Munroe asks crazy hypothetical 
questions, but then proceeds to answer them 
with the perfect balance of science and humor. 


many reminders that although 
advertising yourself in a specific 
niche might seem like a way to lose 
money, it’s actually the ticket to 
making more and to becoming a 
more successful consultant. 

Of course, | read about other 
subjects too. If, like me, you're 
interested in the intersection of 
history, science and food, you should 
read the book Consider the Fork 
by Bee Wilson, which describes the 
evolution (and diversity) of kitchen 
utensils and gadgets. Wonder why 
China and America have different- 
shaped spoons or how forks and the 
shape of our teeth are related? This 
book describes it all. | was introduced 
to this book by the great “Gastropod” 
podcast, which talks about these 
subjects in great depth. 

Another great book that | read this 


year is What If by Randall Munroe, the 


author of XKCD. In What /f, Munroe 
asks crazy hypothetical questions, but 
then proceeds to answer them with 
the perfect balance of science and 
humor. | read the first part of this 


book while eating at a restaurant, 
and | feel bad for the other patrons 
who had to endure me laughing quite 
loudly at the absurdities in this book, 
mixed together with (of course) great 
illustrations and explanations. 

Finally, my children are getting 
older, so | have less and less of a 
chance to read to them. However, 
| did manage to read The Terrible 
Two by Mac Barnett and Jory John, 

a beautiful and funny story about 
pranks (or “hacks”, as we called them 
during my undergrad days at MIT), 
friendship and the all-important goal 
of making a school principal look 
foolish in front of the students. 


Podcasts and Other Resources 
| continue to listen to a very large 
number of podcasts each week, 
partly because | enjoy them so 
much and partly because they're so 
convenient for someone like me, 
who often commutes by bus or train 
to clients’ offices. 

Two new Python-related podcasts 
have emerged in the last year: “Talk 
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Python to Me” and “Podcast.__init__” 
are both interesting, with great 
interviews and many insights into 
technologies that | either use every 
day or should start to learn. 

Other podcasts that | enjoy tend 
to be on the subjects of politics 
and economics—from Slate's 
“Political Gabfest” and “Slate 
Money” to NPR’s “Planet Money” 
to “Podcast for America”, an 
irreverent look at the US political 
race. But there are many other great 
podcasts to try, such as the BBC’s 
“More or Less” about statistics, 
“Functional Geekery” about 
functional programming, “Freelancer 
Transformation” about how to 
become a successful consultant 
and “Startup”, which is all about 
what it’s like to run a venture-based 
Startup company. 

| mentioned earlier my interest 
in helping children learn to code, 
and I've discovered (thanks to my 
ten-year-old son) a terrific site that 
does just that: Code Monkey (at 
http://PlayCodeMonkey.com) uses 
CoffeeScript of all things to teach 
programming to children. However, 
it does so using inventive and 
creative graphics, high-quality error 
messages and a staged approach that 
does a very good job of introducing 
programming to kids aged 8-12. In 
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my son's case, his school paid for 
entry into all of the levels, but | would 
gladly have paid the $30 that Code 
Monkey charges to do all of the levels 
it offers, as well as build some of 
my own. It’s rare for me to say that | 
have seen an educational technology 
that lives up to its hype, but this is 
definitely one of those times. 

Finally, let me mention a site that 
| run called Daily Tech Video 
(http://DailyTechVideo.com), which 
offers a new, high-quality conference 
video every day. If you’re interested 
in open-source technologies, 
programming languages, databases 
and even some computer history, 
| invite you to take a look at my site 
and even to suggest high-quality 
conference talks | might have missed. 
You also can follow the site’s update 
on Twitter at @DailyTechVideo.m 


Reuven M. Lerner trains companies around the world in 
Python, PostgreSQL, Git and Ruby. His ebook, “Practice Makes 
Python”, contains 50 of his favorite exercises to sharpen your 
Python skills. Reuven blogs regularly at http://blog.lerner.co.il 
and tweets as @reuvenmlerner. Reuven has a PhD in 
Learning Sciences from Northwestern University, and he 

lives in Modi'in, Israel, with his wife and three children. 
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Working with 
Command 


DAVE TAYLOR 


Arguments 


getopt—the right way to parse starting arguments in your 


shell script. 


In my last article, | went interstellar 
and created a multi-planet time 
translator, allowing you to learn just 
how old you'd be if you lived on 
Saturn or Pluto. So in this article, 
| want to head back to a more 
fundamental aspect of shell scripting: 
working with command arguments. 

| suspect that most shell scripts 
go through an evolution with 
their command flags, actually 
starting out with none, then maybe 
one or two parsed In a sloppy 
fashion, finally upgraded to a 
proper implementation of getopt, 
and then, perhaps, even being 
completely rewritten as a C++ or 
Ruby program as the complexity 
keeps inexorably increasing. 

The really easy way to parse a 
starting flag, of course, is just to use 
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a conditional statement: 


Lt [ "oa" = Meat i then 
flaga=1 
Fi 


There are a couple problems with 
this approach, however, not the 
least of which is that it ends up 
taking a fair amount of space in the 
code. Because to be proper, the full 
sequence also would include flaga=0 
before the conditional, because 
(repeat after me) you never can 
assume that the shell will correctly 
instantiate the value of a newly 
defined variable. 

The other problem is that after 
this code sequence, the command 
parameters are out of sync: $1 is still 
either possibly the starting flag value 
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Where I always get tripped up is that $# is the 
number of arguments, not the total number of 


words in the command. 


(-a) or another argument or value 
that the user has specified. 

To make this a bit more logical, 
let’s imagine a shell script that's a 
wrapper to something like the terrific 
curl: give it a URL, and it’ll grab 
that content and save it to the local 
directory as a file. Add the imagined 
-a flag, and it'll also include progress 
information. So, a typical usage might 
look like this: 


getpage.sh -a http://ww.1linuxjournal.com/index. html 


In this situation, a number of 
command parameter variables are going 
to be instantiated as the shell script Is 
invoked, notably $# = 2, $0 = 
getpage.sh, $1 = -aand $2 = 
http://www. linuxjournal.com/ 
index.html. 

Where | always get tripped up Is 
that $# is the number of arguments, 
not the total number of words in the 
command. Therefore, if this script is 
invoked without any arguments at 
all, $# seems like it should be 1 (what 
about the command name?), but 


actually it’s 0. 

This goes back to the dawn of 
UNIX development actually, and it’s 
known as the “O index problem”. In 
arrays, the first value is referenced 
as being in slot 0, or array[O]. 

For some developers, that makes 
complete sense, and for others, it 
can be confusing. Indeed, I’ve seen 
C programs where the writer ignores 
the first slot entirely and indexes 
Starting at 1, not 0. 

With that in mind, after the 
conditional has been used to check for 
the -a flag in the first variable space 
($1), what really needs to happen is 
that all the positional variables above 
this one (for example, $2, $3 and so 
on) need to shift down a spot. Ideally, 
after the conditional, $1 contains the 
URL value regardless of whether the 
starting flag was specified. 

That way, the next statement block 
in the script safely can assume that $1 
will be the URL and not have to test 
redundantly to see if it’s still -a. 

This is done with the shift 
command, and so, here's the proper 
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way to test conditionally for an 
optional variable in a shell script: 


flaga=0 

if [ "$i" = "-a" ]; then 
flaga=1 
shift 

Fi 


That works exactly as you’d hope, 
but leads to the next question: what 
happens if the flag has an optional 
value that the user can specify? In the 
case of this curl script, perhaps the 
flag is something like -o output file. 

That's actually an easy addition to 
the above code: 


outputspecified=0 

if [ "$i" = "-o" J]; then 
outputspecified=1 
output filename="$2" 
shift 2 

fi 


As you can see, shift takes 
a single numeric argument that 
reflects how many slots you want 
to have everything cascade. It’s easy 
to see if you consider the positional 
values before and after the above 
code block: 


$ sh getpage.sh -0 test.html SomeURL 
$# = 3 


42 / FEBRUARY 2016 / WWW.LINUXJOURNAL.COM 


$1 = -a 
$2 = test. html 
$3 = SomeURL 


$# = 1 

$1 = SomeURL 
o2°= 

$3 = 


Initially, all three positional variables 
are set, and the arg count ($#) is 3, 
as makes sense given the command 
invocation. But after the shift 2, 
everything's moved down two slots, 
and the arg count also is decremented 
two, as you can see above. 

Now, what if your development 
of the script hits a juncture where 
you realize that it'd be useful 
to have three different starting 
arguments, one of which indeed 
takes an argument? 


Dealing with Lots of Starting Flags 
Even two flags can be a pain, 
because It'd be terrible code that 
would force the user to specify 
them in order, but without that, 
-a, -c and -o could require you to 
test for all three, parse and shift, 
then test for all three again, parse 
and shift, then test for all three 
one more time. That would be 

a nightmare, and | haven’t even 
mentioned how those pesky users 
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The getopt command is going to become your 
best friend if you're building complex user-facing 
scripts, no question. 


have a tendency to combine flags Comparing this statement with the 
too, so would your script properly usage error below, you can see that arg 
parse -ac -0o Or -oc -a too? 1 to getopt is a list of all acceptable 
Enter getopt. flags, with : denoting those flags that 
The getopt command is going to have a required additional argument if 
become your best friend if you're the flag is specified—easy enough. 
building complex user-facing scripts, Now the status variable $? can be 
no question. It basically works by tested: if it’s non-zero, there was an 
extracting all the optional flags and error in parsing the flags, and in most 
parameters, then lets you parse scripts, it’s time to fail out with a 
through them in a uniform manner. usage statement: 
The standard usage is to break 
down combined args with an if [ $2 != 0] ; then 
invocation of getopt, then use the echo \ 
set command to replace the existing "Usage: $(basename $0) {-p SFX} {-n} {-t} PTN NEWPTN" 
Starting flags with the new, neater echo" -n_ sequentially number matching files" 
args, then run through a loop, parsing echo" -p_use specified suffix SFX for filenames" 
them one by one. echo"  -t test only - don't execute resultant cmds" 
Yes, this is easier just to exit 0 


demonstrate, so let me show you the fi 
command flag parsing segment from 


a different production script. This You'll often see scripts that have 
particular command has three possible the usage sequence pushed into 
Starting flags, -n, -p and -t. a separate function to keep the 

The first step in the script is to have code clean. Also, note the use of 
getopt normalize whatever the user $(basename $0) in the first echo. 
has specified: That's a handy trick to compensate 

for the fact that most of the time $0 

args=$(getopt np:t $*) is going to be the full name of the 
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Now all the positional parameters are neatly 
organized and ready to parse, something that’s 
traditionally done with a case statement wrapped 
in a for loop (no difference from an enigma 
wrapped in a dilemma, of course). 


script, including path. So tapping 
basename is just for aesthetics! 

Finally, the statement that does the 
real work: 


set -- $args 


Now all the positional parameters 
are neatly organized and ready to 
parse, something that’s traditionally 
done with a case statement wrapped 
in a for loop (no difference from an 
enigma wrapped in a dilemma, of 
course). It looks like this: 


for i 

do 
case "$i" 
in 


-n ) renumber=1 ; shift ;; 


-p ) fixpng=1 ; sfx=$2 ; shift 2 ;; 
-t ) doit=0 * Share 37 
esac 


done 


| have a particular style with the 
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semicolons in my case statements, 
mostly just to ensure that | use 
the needed ;; sequence to 
terminate each of the individual 
conditionals, but otherwise, it 
should be easy to understand. 

The only thing missing from this 
code fragment is something | alluded 
to earlier. What Is it? 

And, that’s it for this article. Now, 
go back to your latest shell script, and 
just for practice, go ahead and add 
some optional starting flags and parse 
them with getopt.m 


Dave Taylor has been hacking shell scripts since the dawn of the 
computer era. Well, not really, but still, 30 years is a long time! 
He’s the author of the popular Wicked Cool Shell Scripts and 
Teach Yourself Unix in 24 Hours (new edition just released!). 

He can be found on Twitter as @DaveTaylor and more generally 
at his tech site: http://www.AskDaveTaylor.com. 
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The Powers 


SHAWN POWERS 


When the lights go out, do your servers too? 


I live in an old house with old 
wires in an old town with a dated 
infrastructure. I’ve never really 
considered the quality of the 
electricity being pumped into my 
house, but since we moved to this 
charming house loaded with character 
(that’s the nice way of saying “old”), 
it’s something I’ve had to deal with 
often. | made a lot of assumptions and 
even made some expensive purchases 
that in hindsight were silly. So for this 
article, | figured | should talk about 
power, because if you’re reading 
Linux Journal, you're familiar with 
things that plug in to the wall. 


The Symptoms 

The biggest red flag for me in the new 
house was my Internet connection 
behaving poorly. I've literally had the 
technicians out here more than 
two-dozen times trying to track down 
why | have massive packet loss. At 
first it was unusable (50% packet 

loss is something TCP/IP can’t even 
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error-correct), but after months of 
struggling, | have only minimal loss. 
Unfortunately, my day job requires me 
to have a solid Internet connection, 
so I’ve been forced to install two 
separate connections and provide 
failover support if (when!) one of 
them goes down. Although the 
Internet connections may be the most 
sensitive to electrical issues, | know 
that unstable voltage and noise can 
be bad for all my servers as well. 
Quite honestly, even if you're not 
having Internet issues, there are 
some things to look for that might 
be symptoms of power problems. 
Now that I’m paying closer attention, 
| notice that a few times a day, 
our lights flicker or quickly dim. | 
assumed at first it was when a large 
motor kicked on (causing a sag) or 
something like that, but now | don’t 
think it Is. It appears to be unrelated 
to anything in the house, and it 
happens often enough that it’s not 
just a fluke. If you notice tiny glitches 
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I've actually never had problems with EMI/RFI 

on server equipment, but if you have any audio 
equipment and you hear a buzz, or odd noises in 
your recordings, a line conditioner might really help. 


in the Matrix, it might be a sign that 
your power fluctuates. 


The Minimum Protection 

Even if your house’s electrical service 
is rock-solid, it’s very important to 
have your electronic equipment on 

a surge protector. Although they 

used to be expensive, you can get a 
really nice one for less than $20 at 
most stores. A surge protector’s job 

is to protect your equipment from 
current fluctuations, but only extreme 
fluctuations or surges (thus the name). 
A surge protector won't generally help 
with minor issues like those in my 
house, but during a lightning storm, 
you'll be glad you spent the money. 

If you live in a place that rarely gets 
lightning storms, or if you just don’t 
think a lightning strike is likely in 
your area, | still highly recommend a 
surge protector, or surge suppressor 
as they're often called. When | worked 
at a local school district, there was a 
failure on one leg of the three-phase 
power feed that caused a huge surge 


to enter the building. It was a sunny 
day without lightning, and still the 
surge destroyed thousands of dollars 
worth of non-protected equipment. 


Conditioner, Not Just for Hair 

The local cable technician 
recommended | purchase a power 
conditioner in order to give my 
equipment clean power. This actually 
started my research on power, and 
before | did much studying, | ordered 
a rack-mounted power conditioner 
for all my servers. | wish | had done 
more research before ordering it, but 
it does serve a purpose, and what it 
does, it does very well. 

A power conditioner does a couple 
things. It protects your equipment 
from surges. In fact, a surge protector 
is a power conditioner of a sort; it’s 
just the most basic kind. The other 
main purpose of a power conditioner 
is that it “cleans” the power signal 
by filtering out electromagnetic 
interference (EMI) and radio frequency 
interference (RFI) that can really mess 
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Figure 2. | don’t regularly look at the front panel, so really this readout is sufficient. | just prefer the 


large LCD numbers showing incoming voltage. 


with sensitive equipment. I’ve actually 
never had problems with EMI/RFI on 
server equipment, but if you have any 
audio equipment and you hear a buzz, 
or odd noises in your recordings, a 
line conditioner might really help. 

The model power line conditioner 
| bought has a really cool display 
on the front that shows the line 
voltage. In fact, although it doesn’t 
do anything to help with voltage 
fluctuations, it is nice to have the 
readout showing me as voltage 
comes and goes. Figure 1 shows 
my power conditioner. It’s in my 
office now, but | can see the 
voltage vary anywhere from 103V 
to 124V throughout the day. That’s 
a huge variance, and it’s very likely 
what has been causing part of my 
Internet problem. 
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The Big Guns 

Although the power line conditioner 
is great at identifying voltage 
issues, it doesn’t actually do 
anything to fix them. Unfortunately, 
the tool to normalize voltage is 
quite expensive. | paid around 

$400 for a rack-mounted voltage 
regulator. The good news Is that 
most voltage regulators not only 
regulate the voltage, but they also 
condition the line and protect from 
surges. (But you really should read 
the fine print to make sure.) 

The voltage regulator on my server 
rack functions just like a power strip. 
It plugs in to the wall, and then | 
plug my devices in to the back. All 
the devices see a perfectly steady 
120V power source, however, even if 
my actual house voltage gets wacky. 
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It’s not a magic device, and it won't 
work in the case of a brown out, but 
it can handle voltage sags down to 
95V, which is lower than I’ve ever 
seen it drop in my house. 

The specific voltage regulator | 
purchased is the Furman M-8X AR 
(Figure 2). It supports up to 15 amps 
of service, which is all my server 
rack power line is rated for anyway. 
| do wish it had a clearer readout 
for incoming voltage like my line 
conditioner does (the line conditioner 
is the Furman M-8Dx), but that 
doesn’t affect its performance, just 
my desire for data! 


Sometimes, the Voltage Is Zero 
That last thing | want to touch on 
here is a battery backup. There's a 
lot to be said regarding the type of 
battery backup you use, but | just 
cover them briefly here. 

Standby UPS: This is the type of 
UPS most folks use, even in server 
rooms. There is a battery, which is 
kept charged by the main power 
line, and if the power goes out, it 
quickly powers an inverter to supply 
120V current. This is very efficient, 
because the inverter doesn’t 
constantly convert the battery power 
into AC, it only “does work” when 
the power goes out. 

The downside of all this efficiency 


is that there’s a small cutover time 
when the line power is disrupted 
before the inverter takes over. It is 
usually measured in milliseconds, 
and the more expensive the standby 
UPS, the quicker the cutover time. 
This is typically not a problem 

for most computer hardware, but 
particularly sensitive computers may 
reboot during switchover. 

Standby-Ferro UPS: This type 
of UPS functions in the same way 
as a Standby UPS, except there is a 
built-in transformer that regulates 
the voltage and provides a bit of 
a “buffer” during the cutover to 
inverter power. Since transformers 
use magnetism while converting 
voltage, there is a residual magnetism 
that usually smooths over the 
transition so that even sensitive 
equipment doesn’t suffer. 

Since they have built-in voltage 
regulation, they help with fluctuating 
voltage even when there’s not a 
power outage. Unfortunately, they 
are known to overheat with some 
generators, especially those not 
generating pure sine waves. Still, with 
the buffering effect of the voltage 
regulation circuitry, Standby-Ferro 
units are usually preferred and usually 
cost significantly more than standard 
Standby UPS units. 

Line Interactive UPS: These types 
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of backups provide the smoothest 
transition during an outage, because 
they constantly use their inverter to 
produce electricity. The only switchover 
is whether the inverter is taking power 
from the batteries or the wall current, 
and since both are connected, there's 
not even a millisecond of cutover time 
to worry about. 

These are the most expensive types 
of UPS devices, and although they're 
very nice for sensitive equipment, 
they're often overkill for protection 
from the occasional outage. Still, if 
you're looking for the best of the 
best and price is no object, a Line 
Interactive UPS is the Rolls Royce of 
the UPS world. 


50 / FEBRUARY 2016 / WWW.LINUXJOURNAL.COM 


My Cheap, Massive Solution 

| have terrible luck with UPS devices. 
It seems like the batteries are always 
dead when our power goes out, 

and | have no idea until it’s too late. 
And then, even the fairly expensive 
rackmount units are often disposable 
without the ability to change their 
batteries. Even if the batteries are 
replaceable, and you actually know 
they've gone bad, UPS devices don’t 
last very long. Usually the best you 
can hope for is to keep your servers 
up long enough that they can shut 
down properly instead of just losing 
power. That's a great benefit, but it’s 
frustrating if you want to function 
during an outage. 


Figure 3. | really like having a separate inverter/ 
charger and battery. If the battery dies, | 

can replace it without replacing the inverter 
hardware itself. Plus, solar power will be fun to 
play with! 
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For my home, | decided to 
separate the inverter from the 
battery. | bought an AIMS inverter/ 
charger unit (Figure 3) that is hard- 
wired inline with my house wiring. 
The specific unit | bought is the 
3000-watt pure sine inverter. It gives 
clean voltage and uses external 
batteries. In my case, it uses a huge 
12v marine battery. This will keep 
my little server rack running for 
quite some time before the battery 
is drained, and since it’s a robust 
lead acid battery, it seems to last 
better than the tiny rackmount UPS 
batteries. Time will tell. 

| decided to go with 3000 watts 
because | wanted to make sure 
the full 15 amps of service would 
be supported, even though | don’t 
use nearly that much on my rack. 
| also like the AIMS power inverter 
because it comes with connections 
that support charging with a solar 
array along with grid power. | have 
no intention of going “off grid” 
with my unit, but | love the idea of 
offsetting a little of my electrical bill 
with solar panels. (That will likely be 
a future article!) 


Probably More Than You Need 
Quite honestly, my power problems 
are likely more pronounced than most 
folks. In our last house, | never had 
any problem with our line power at 
all, and the idea of line conditioners 
and voltage regulators seemed absurd. 
With our new house, however, if | 
want to be able to work from home 
and use computers all day, | have to 
make sure my power is rock-solid, 
and my two Internet connections are 
reliable. But who knows, maybe | 
actually will decide to go off the grid 
someday. If that ever happens, I'll be 
sure to write about solar panels and 
battery banks. Until that time comes, 
enjoy your conditioned electricity! m 


Shawn Powers is the Associate Editor for Linux Journal. 

He’s also the Gadget Guy for LinuxJournal.com, and he has an 

interesting collection of vintage Garfield coffee mugs. Don’t let 
his silly hairdo fool you, he’s a pretty ordinary guy and can be 
reached via e-mail at shawn @linuxjournal.com. Or, swing by 

the #linuxjournal IRC channel on Freenode.net. 


TEE 
Send comments or feedback via 
http://www.linuxjournal.com/contact 
or to ljeditor@linuxjournal.com. 
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UNDER THE SINK 


Fast Network 
Routing, Meet 


COLUMNS 


SUSAN SONS 


Userspace 


Interview with Katerina Barone-Adesi about developing 
with the Snabb Switch network toolkit, working outside 
the Linux kernel for cleaner code and faster throughput. 


Recently, | had the pleasure of 
sitting down with Katerina Barone- 
Adesi, a developer from the Snabb 
Switch community to talk about 
Snabb Switch, writing network 
infrastructure code and more. In case 
you aren't familiar with Snabb Switch, 
it is described in its README as “a 
simple and fast packet networking 
toolkit”. The thing that caught my 
attention was Snabb Switch’s unusual 
architecture: it’s small and fast, but 
written largely in LuaJit, which | didn’t 
expect. It circumvents the kernel 
almost entirely, preferring to interface 
directly with network hardware and 
network code that is slower than that 
in Snabb Switch. 


SS: Can you give me some 
examples of what Snabb Switch 


can be used for? 
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KB: Snabb should be interesting 

to anyone who wants to do high- 
speed networking with the flexibility 
of writing software and seriously 
needs to consider several gigabits a 
second or more of traffic. Secondarily, 
it is interesting to hobbyists and 
researchers. Alexander Gall, at 
SWITCH, wrote an L2VPN on Snabb, 
because the features he wanted were 
not available elsewhere. It is a really 
powerful tool for individuals and small 
teams who need to do unusual or new 
things with networks. 


SS: So, how did Snabb Switch 
get started? 


KB: Snabb was started by Luke Gorrie, 
who has a fairly deep background 

in networking, along with a 
professional history using Smalltalk, 
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Forth, etc. He was influenced by 
some of Alan Kay’s ideas about 
building small, comprehensible 
systems that individuals can 
understand—and also by the desire 
to do flexible, high-performance 
software-defined networking on 
commodity x86 hardware. 

It started with a code budget—the 
binary should fit on a floppy, compile 
in less than a second (and less than 
a minute including dependencies), 
and be less than 10,000 lines of 
code. It has grown a bit beyond 
that. Essentially, it lets people do 
more or less line speed networking 
using 10 gigabit cards, on a single 
core, without the overhead of using 
the kernel. The kernel is involved 
peripherally (for instance, it uses 
hugepage support), but Snabb 
implements its own drivers, and the 
cards it is handling are not managed 
by the kernel. 


SS: A lot of what's out there in this 
space today is big...when you're 
introducing people to these tools 
(either building them or using them) 
do they usually get where you're 
going right away? If not, what are 
the parts that are hardest for them to 
wrap their heads around? 


KB: | would say the biggest initial 
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hurdle, assuming some knowledge 

of networking, is that Snabb is a 
fairly young project. Some parts are 
well documented, others are not, 

and there are not many examples of 
how its libraries work, even scattered 
around GitHub. Snabb itself is pretty 
small and well factored, which helps 
a lot. More generally, networking 

is a vast space. When it comes to 
introducing people to using tools 
built on Snabb, there is often some 
incredulity about using a garbage- 
collected high-level language to do 
high-performance networking, but 
LuaJIt has a solid track record in this 
space by now. Aside from that, most 
people are not used to thinking of the 
timescales involved. Depending on the 
expected average packet size, if you 
are driving a 10 gigabit card, you have 
tens to hundreds of nanoseconds per 
packet. With these constraints, you 
need to think about when you will hit 
RAM, and things like system calls are 
prohibitively expensive. 


SS: Why did you end up choosing 
LuaJIt instead of, for example, 
Straight C? 


KB: Snabb is written in LuaJit, so the 
choice of language (and license— 
it’s APLv2) came with the choice to 
use Snabb. That said, using LuaJlt is 
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excellent for projects like this. It is 

a just-in-time trace compiler, so the 
actual code being run is shaped by 
the packets being seen, and there 
are no branch mispredictions within a 
trace. It also makes it extremely easy 
to use C when appropriate. It has 

an extremely nice FFl, good support 
for packed structs, and calling C 
code seems as easy as It would be 

if the whole project were in C. The 
code itself tends to be more concise 
and quicker to write—it is quite 
comparable to Python or Ruby in that 
regard. It is quite pleasant not to have 
to read past boilerplate details. 


SS: What tools are you using to 
track down bottlenecks and get the 
optimization you need for the tight 
time constraints? 


KB: The tools that we are using 

to track down bottlenecks include 
PMU counters and a combination 

of logical thinking about what is 
likely to be a bottleneck, followed by 
measuring changes to see whether 
they actually make an improvement. 
Having a small system makes the 
latter a surprisingly tractable 
approach. We also look at LuaJIt’s 
trace dumps, which include IR 
(Intermediate Representation) and 
x86 assembly. 
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SS: Have you had to break parts out 
to C yet? 


KB: | have not personally had to 
break parts out to C yet. People 
occasionally do—for instance, if they 
want to hide an unpredictable branch 
from the trace compiler. Another 
alternative that a colleague of mine 
used recently is DynASM, which 
allows you to generate assembly 
dynamically. He used it for some 
AVX2 instructions to access memory 
256 bits at a time. One of the very 
surprising things about LuaJlit is that 
it often is as fast as C, and swapping 
code out for C blindly can genuinely 
make performance worse, as you lose 
the benefits of traces. 


SS: How much do you have to 
know about the specific hardware 
this will be running on to keep 
the speed up? Is architecture plus 
network card enough, or are you 
falling into “oh, sorry, you have a 
brand of RAM we never tested...” 
level of optimization specificity? 


KB: So far, micro-architecture 
(Haswell vs. Sky Lake) and raw CPU 
speed do make a noticeable difference 
in performance for some applications. 
For others, like the packetblaster 

app that ships with Snabb, any 
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modern CPU Snabb can run on is so 
far beyond what is needed that the 
bottleneck is the network cards. 


SS: So, to recap, Snabb Switch Is an 
open-source userspace application 
written mostly in LuaJIt that’s pushing 
network traffic on run-of-the-mill 
COTS x86_64 easily at 1Gbps, with 

a 100Gbps target in 2016...by 
bypassing the Linux kernel for faster 
performance...and you say you're 
getting comparable performance to 
C code from LuaJit, without excessive 
scaffolding for performance testing. 


KB: Yep. 
SS: So, how big is this thing again? 


KB: The source is less than 5MB; the 
resulting binary is something like 2MB 
before compression. 


SS: Impressive. However, this isn’t 
just some endpoint software, it’s 
infrastructure—what does your 
team do to ensure code quality 
and security? 


KB: For code quality, my team uses 
continuous integration, tests and 
code review. All code is reviewed by 
at least one person other than the 
author before being committed to 


the main development branch. We 
have a variety of end-to-end and unit 
tests. Pflua also has property-based 
tests, which compare its results with 
and without optimization to libpcap’s 
results on randomly generated filters. 
We have plans to do more property- 
based and fuzz testing. Additionally, 
the small size of the system makes it 
easier to pay close attention to code 
quality. For security, all of the above, 
along with thinking hard and paying 
close attention to RFCs and their 
notes on security. Our fragmentation 
reassembly code rejects overlapping 
fragments as more recent RFCs 
recommend, for example. 


SS: When you bring on new 
developers, how do you inculcate 
them into this kind of rigor given how 
little it's present in other programming 
shops, and how essential it is for 
developing infrastructure software? 


KB: | introduced it to my team when 
| joined in 2014. We started using 
continuous integration software 

and property-based testing at that 
point, both of which significantly 
increased the reliability of pflua. 
Code reviews also help. We have had 
only one person join our team since; 
as we grow more, perhaps | will 
have a better answer for that. The 
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code also had a lot of conceptual 
rigor to begin with. My colleague 
who shaped pflua most deeply, Andy 
Wingo, spends most of his time 
working on compilers, which also 
demands a fairly high level of rigor 
and coherent design. 


SS: If you were to guess, about what 
percentage of your project's effort is 
spent on new code/features vs. testing/ 
refining/optimizing, writing test code 
and scaffolding for fuzz testing vs. 
documenting and planning tasks? 


KB: It depends on the phase of each 
project, I'd say. They tend to start off 
with a flurry of features and planning, 
and some testing is done in parallel. 
Later in each project, we optimize and 
test more. The efforts compliment 
each other well, as more extensive 
tests catch subtle mistakes attempts 
at optimization can introduce. Some 
of the features are specifically linked 
to performance goals, which makes 
them fall thoroughly into being 

both new code/features and testing/ 
refining/optimizing at the same time. 
My colleague Andy Wingo’s recent 
work with dynasm is an example. 
Earlier in the lwaftr project, almost 

all of the time was going to new 
code and features, with perhaps 20% 
going into testing them. Before the 
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alpha release, the whole team focused 
almost entirely on performance for a 
few weeks. At the moment, | think it 
might be around 50/50. 


SS: If you were to break it down 
to two to five principles the Snabb 
Switch toolkit is built on, what 
would they be? 


KB: | would actually break it down to 
just one: systems should be small, and 
individuals should be able to hold the 
whole system in their head. Most of 
the other principles are consequences 
of this, and the high-speed software- 
defined networking/network function 
virtualization niche it is designed for. 


SS: Why is this so important to Snabb 
Switch in particular? 


KB: It is an important principle for 
software wherever it can reasonably 
be applied, and it is one thing that 
strongly drew me toward Snabb. | 
can only speculate that for Snabb 

in particular, it came from Luke's 
background with Smalltalk—where 
he took the quotes describing this 
principle from—and other minimal 
systems like Forth. 


SS: What do you like best about 
working on Snabb Switch? 
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Between the design principles and being able not 
only to replace old network functions but also 
implement entirely new ones on commodity 

x86 hardware and network cards that cost a few 
hundred bucks, I think it has a really exciting future. 


KB: Snabb is a really fun and 
interesting project to be involved 

with these days. Between the design 
principles and being able not only 

to replace old network functions but 
also implement entirely new ones on 
commodity x86 hardware and network 
cards that cost a few hundred bucks, | 
think it has a really exciting future. 


SS: So, what got you interested in 
tweaking this type of networking code? 


KB: This is my day job. | joined Igalia 
in 2014, when the networking team 
was working on pflua. Pflua is a 
library on top of Snabb that uses a 
subset of libpcap’s filtering language. 
It is smaller, faster, and the subset 
that it supports is believed to be 
entirely compatible except for libpcap 
optimizer bugs. 

| got into networking as a hobby 
back around 2000 though. | ran a small 
home network on Linux, various BSDs, 


read a lot of books, played around, 
etc. | have liked systems programming 
and dynamic languages for about as 
long, so a job that involves all of these 
elements is pretty fun. 


SS: Did you make a specific effort to 
choose companies where you'd be 
doing open-source work, or was it luck? 


KB: Igalia does only free software, 
and this was a major factor in my 
choosing to work for them. | did also 
interview at other companies that 

do a mixture of free and non-free 
software. 


SS: Do you have any advice for 
coders interested in Snabb Switch 

in particular, or in moving from 
places higher up the stack into more 
infrastructure-y areas of programming 
work like what you're doing? 


KB: For coders interested in Snabb 
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Switch in particular, it depends on 
their background; ones with more 
relevant experience can probably jump 
in and implement something Luke 
brain-dumps about on the mailing list, 
or something else entirely. For those 
newer to these kinds of programming, 
| would recommend starting by 
writing a Snabb app. These can be 

as simple as a packet blaster or an 
app that echoes packets between 
interfaces, or more complex than the 
I2vpn or IWAFTR, and can be built 

up incrementally. Some of the tests 
can be run, and app development 

can be done, on any modern x86 
machine running Linux. | prototyped 
the IWAFTR on my development 
laptop, which has no Ethernet cards. 
Snabb has a mechanism for plugging 
apps into each other that makes 


Resources 


running it on real hardware a simple 
matter of changing a couple lines of 
configuration and recompiling. 


SS: Thanks again, Katerina, for 
sharing your time and expertise.m 


Susan Sons serves as a Senior Systems Analyst at Indiana 
University's Center for Applied Cybersecurity Research 
(http://cacr.iu.edu), where she divides her time between 
helping NSF-funded science and infrastructure projects 
improve their security, helping secure a DHS-funded static 
analysis project, and various attempts to save the world 

from poor information security practices in general. Susan 
also volunteers as Director of the Internet Civil Engineering 
Institute (http://icei.org), a nonprofit dedicated to supporting 
and securing the common software infrastructure on which 
we all depend. In her free time, she raises an amazing 
mini-hacker, writes, codes, researches, practices martial arts, 
lifts heavy things and volunteers as a search-and-rescue and 
disaster relief worker. 


Snabb Switch: https://github.com/SnabbCo/snabbswitch 


“Snabb Switch Deep Dive on Software Gone Wild” by Ivan Pepeinjak: 
http://blog.ipspace.net/2014/09/snabb-switch-deep-dive-on-software-gone.html 


“L2VPN over IPv6 with Snabb Switch on Software Gone Wild” by Ivan Pepeinjak: 
http://blog.ipspace.net/2014/12/l2vpn-over-ipv6-with-snabb-switch-on.html 


“High-performance packet filtering with Pflua”: 


https://archive.fosdem.org/2015/schedule/event/packet_filtering_pflua 


“Deutsche Telekom TeraStream: Designed for Simplicity” by Ivan Pepeinjak: 
http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.html 
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Linux Journal eBook Series 


GEEK GUIDES 


Practical books for the 
most technical people on the planet. 


—_ 
Downloar 
Now 


Agile Product 
Development 


Author: Ted Schmidt Sponsor: IBM 


Every day, a new Internet-connected gadget hits the market, 
Development promising to make everyone’s lives incrementally more 

convenient. Devices like Fitbit and Sen.se Mother help monitor 
your health. Trackdot and Bikn let you track your possessions 

through GPS. Delphi Connect, Nest, SkyBell and LIFX all make 
it possible to control home appliances from anywhere. By 2020, Gartner forecasts 
that there will be more than 25 billion of these “things” connected to the Internet. 
And every day, consumers are demanding more from these things: more features, 
more interconnectivity, more convenience. 


vowcu Improve Business 


” sk Vo 


GEEK GUIDE Processes with an 
Enterprise Job Scheduler 


Author: Mike Diehl Sponsor: Skybot 


Modern IT shops run a whole lot more than just a few file and 
print servers like they did in the old days. Today’s enterprises 

are vastly more complex, often with servers in different data 
centers and even scattered all over the globe. Additionally, they typically have 
Windows servers, Linux servers, mainframe servers and various other flavors of UNIX 
servers, and all of these servers, all over the network, produce and process data. 


Go to http://geekguide.linuxjournal.com 


NEW PRODUCTS = 


SoftMaker Office 


Founded in 1987, the Germany-based software vendor SoftMaker 

has spent the modern PC era developing and improving its own office 
program, SoftMaker Office. SoftMaker Office 2016 for Linux is SoftMaker’s 
latest release of the office suite, which consists of the word processor 


TextMaker, the spreadsheet program PlanMaker and the presentation 

program SoftMaker Presentations. SoftMaker’s value proposition vs. other office suites includes 
a compact footprint, stability, availability in Standard and Professional versions, add-ons for the 
Thunderbird e-mail client, as well as compatibility with (and a five-fold soeed advantage over) 
Microsoft Office. The new SoftMaker Office 2016 boasts more than 400 new features, such 

as optimized software architecture accelerated by the use of OpenGL for screen display and 
multiple CPU cores for graphic rendering, improved compatibility features with MS-Office, 
built-in version management for documents, support for 9,000 label formats, improved pivot- 
table functionality, smart guides for alignment of objects in presentations and more. The 
Professional edition adds four high-quality dictionaries by Berlitz for easy translation between 
English, Spanish, Italian, German and French. All modern Linux distributions are supported. 
http://softmaker.com 


Ziften ZFlow 


| 
ZITTEN 
~ = a As enterprises have migrated to the cloud for economic and 
operational purposes, traditional network visibility has been lost 
due to a lack of access at the infrastructure layer. To help organizations avoid operating in the 
dark in the penguin zone, security specialist Ziften has released a new version of its flagship 
Ziften ZFlow for the Linux operating system. Ziften ZFlow delivers greater network visibility 
by providing full visibility, contextual intelligence, user behavioral analysis and integration 
into previously deployed security tools. This new integration with Linux provides previously 
non-existent visibility into the public cloud infrastructure and enables Ziften’s new Cloud 
Visibility Initiative. The initiative helps secure cloud operations with the visibility that security 
professionals need to identify and respond to potential threats and attacks quickly. Ziften 
adds that ZFlow is lightweight, meets IPFIX standards and enables better east-west visibility to 


identify lateral movement of an attack within the data center. 
http://ziften.com 
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NEW PRODUCTS 


Thomas Erl, Wajid Khattak 
and Paul Buhler’s Big Data 
Fundamentals (Prentice-Hall) 


Big Data Fundamentals Because Big Data will only get bigger, the case for reading Big Data 
Fundamentals: Concepts, Drivers & Techniques will only get stronger. 
Co-authored by best-selling IT author Thomas Erl and collaborators 
Wajid Khattak and Paul Buhler, Big Data Fundamentals is a pragmatic, no-nonsense introduction 
to Big Data for business and technology professionals. The authors clearly explain key Big Data 
concepts, theory and terminology, as well as fundamental technologies and techniques. All 
coverage is supported with case-study examples and numerous simple diagrams. The authors 
commence by explaining how Big Data can move an organization forward by solving a 

wide range of previously intractable business problems. Next, they demystify key analysis 
techniques and technologies and show how a Big Data solution environment can be built 
and integrated to offer competitive advantages. Other key topics include differences between 
Big Data and previous forms of data analysis and science, business motivations and drivers 
behind Big Data adoption and recognizing the five “V” characteristics of datasets in Big Data 
environments: volume, velocity, variety, veracity and value. 

http://informit.com 


Concepts, Drivers & Techniques 


Varnish Software’s Zipnish = = 
Gaining insights into how quickly services are it 

running or whether they are adding latency is a — . FS} 
difficult task in distributed architectures such as 
microservices. To simplify this task, Varnish Software if 

launched Zipnish, a new open-source, architecture- sal a, 
agnostic tool that tracks performance and helps 


resolve latency issues in microservices architectures. Zipnish uses the Varnish logging API from 
Varnish Cache 4.0 to monitor transactions and uses Python and the event library Twisted to 
transport the data. MySQL is used as the database for storage. The presentation back end is 
done in Python, whereas a slightly modified version of Zipkin is used as front end. 
http://varnish-software.com 
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Red Hat CloudForms S) redhat. 


To meet customer demands in today’s hybrid, heterogeneous world, Red Hat established an 
alliance with Microsoft to release Red Hat CloudForms 4, the latest version of the company’s 
open-platform hybrid cloud management solution that now features Microsoft Azure support. 
Red Hat declares that Red Hat CloudForms 4 goes beyond self-service, offering a consistent 
experience and comprehensive lifecycle management across platforms, covering virtualization, 
private cloud, public cloud and containers. In addition to enabling Azure customers to manage 
those workloads and resources within CloudForms, release 4 adds management for container 
architectures and improved self-service dashboards and charts to better analyze the relationships 
between different cloud platforms and container hosts. Other platforms besides Azure into 
which Red Hat CloudForms 4 provides operational insight include Amazon Web Services, 
Hyper-V, OpenShift by Red Hat, OpenStack, Red Hat Enterprise Virtualization and VMware. 
http://redhat.com 


Imagination Technologies’ Creator Ci40 


“Where do | start?” is a common question for those of us exploring 
the Internet of Things (lol) and embedded computing markets. 
Imagination Technologies answers that question in the form of a 
product, namely the firm’s new Creator Ci40 loT kit, which helps one 
make sense of the oh-so-many boards and accessories, hardware 

and software interfaces, connectivity standards and APIs in loT. The 
Creator Ci40 includes not only the hardware building blocks needed 
to prototype a wireless loT system from scratch quickly, but also the open-source software 
frameworks, the network stacks and the cloud connectivity capabilities required to connect 
and authenticate devices to the cloud securely. The included Ci40 microcomputer has been 
specifically designed for smart home, loT and other connected devices (drones, robots and so 
on) and features the hardware requirements (802.1 1ac 2x2 Wi-Fi, an Ethernet port, SD and 
USB storage) for use as a high-speed wireless router. The board runs Linux (Debian, OpenWrt) 
and is part of Google's golden reference program for its Brillo OS for loT. Rounding out the 
kit are add-ons from Mikroelectronica: two battery-powered Clicker board development kits 
and three Click boards for measuring temperature, detecting motion and controlling a relay. 
http://imgtec.com 
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NEW PRODUCTS 


Opera Max 


Cheapo data hogs (like you) with limited data plans can stretch their 
budgets and double their streaming of video—and now music—using Opera Max, Opera ASA‘s 
upgraded data-management and data-saving app for Android. The latest release adds streaming 
audio optimization to the existing video optimization functionality. Opera notes that nine hours 
of streaming music or on-line radio require 1GB of throughput, which easily can burn through a 
data plan when using mobile networks. Complementing Opera Max's existing ability to optimize 
streaming-video services like YouTube and Netflix comes new functionality that optimizes 
streaming music apps, such as YouTube Music, Pandora, Slacker Radio, Gaana and Saavn, 

with others on the way. Opera's optimization technique for audio involves applying technology 
from Rocket Optimizer for converting MP3 and MP4 audio streams to the more efficient AAC+ 
codec, which delivers high audio quality over a low bitrate connection to any compatible device. 
http://opera.com 


Sander van Vugt’s Beginning the Linux 
Command Line, 2nd Edition (Apress) 


IT author Sander van Vugt says that his book Beginning the Linux 
Command Line, now in it second edition, is for everyone who uses 
Linux—no exceptions. Van Vugt’s book is a guide to Linux in which 
the user digs deeply into the system, wielding the shell as a shovel. 
Key overarching goals include understanding how Linux Is organized 
and how to “think Linux”. Other learning objectives include finding the right command 

for the task at hand, working with text editors and intelligent filters, shell programming, 
configuring access to hardware devices and more. Beginning the Linux Command Line is 
checked against all of the most important Linux distributions and follows a task-oriented 
approach that is distribution-agnostic. This new second edition of the book covers features 
of the very latest versions of the Linux operating system, including the new Btrfs filesystem, 
the systemd boot procedure and firewall management with firewalld. 

http://www.apress 


Please send information about releases of Linux-related products to newproducts@linuxjournal.com or 


New Products c/o Linux Journal, PO Box 980985, Houston, TX 77098. Submissions are edited for length and content. 
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Temperature Control in a Homebrewing Tun Using a BeagleBone Black 


TEMPERATURE 
CONTROLIN A 
HOMEBREWING 
TUN 


Using a BeagleBone Black 


love beer—good beer, not 
industrial beers. The ones 

| brew myself, where | can 
control the ingredients and the 
process, are very good. 

Beer brewing is a simple process—it 
used to be a duty of housewives on 
the farms here in my country long 
ago. The process | and my two brew 
mates use is a bit more complicated 
than the process used in ancient 
country houses though. One thing we 
can do nowadays is mash the malt 


in several temperature stages. This 
results in a more complex beer with a 
more malty body. And, we now have 
better efficiency—that is, more sugars 
for the yeast to produce alcohol from, 
meaning less malt will provide the 
same amount of alcohol in the beer if 
our efficiency is higher than normal. 
But, some problems arise when 
trying to brew using our simple 
brew tun. The tun is a big pot with a 
29-liter capacity. The pot has a built-in 
heating element with a rated power 


Figure 1. 
Simplified 


Homebrew Tun 


Flow 
direction 


BeagleBone 
Black 
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of 1800 W (Figure 1). Helped by a 
circulating pump, the mash (fluid) 
circulates from the bottom of the tun 
to the top overflowing the malt. 

Many of our beers have a starting 
point between 45°C-55°C. In this 
temperature range, we are preparing 
the proteins in the malt for the later 
process of extracting sugars out of 
the malt. The optimal temperature 
for the beta and alpha enzymes that 
convert starch to sugars is between 
63°C-68°C. So after the protein 
stop in the 50°C range, we have to 
raise the temperature in the mash. 
But unfortunately, too much heat will 
burn some of the particles floating 
in the mash onto the heater in the 
bottom of the tun, resulting in a bad 
taste and difficulty getting the heat 
from the heating element into the 
mash. We have determined empirically 
that from 59°C-60°C we can apply 
full power. To finish the process, the 
temperature is raised to 78°C to stop 
the enzymes from working. 

So, we need a way to control the 
amount of power fed into the heating 
element. What is more natural than to 
use a BeagleBone Black to control the 
rate of energy burned in the heater 
through a Pulse Width Modulated 
(PWM) controller already built in to the 
BeagleBone? The PWM will produce 
a pulse of varying width, turning a 
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relay on and off in controlled intervals, 
thereby controlling the amount of 
power fed into the heater. If the relay 
is closed for 2 seconds and off for 8 
seconds, 2/8 * 1800 W = 450W is fed 
into the heater. Well, to be honest, this 
problem could be solved using a simple 
555 chip and a potentiometer, but 
since we are programmers, we tend to 
pull out the tools we love, even when 
it’s overkill. 

A side benefit from setting up 
a BeagleBone is the ability to log 
temperatures during the whole brew 
process. This is handy for evaluating 
the beer when adjustments to the 
recipe, which include the processing, 
are discussed. Previously, we noted 
temperatures and times on a piece 
of paper a few times during the 
brew process—when we remembered 
to do it. Another benefit is that the 
whole thing can be controlled from a 
Web page. 


System Overview 

I've already mentioned the 
requirements, and from there, you 

can extract that we need a controlling 
functionality that will read the 
temperature in the mash when it leaves 
the bottom of the tun. Based on the 
temperature read, it will set the period 
time of the output PWM. Also, we 
need a database enabling us to log the 


collected data during the brew process. embedded a Websocket server into 


Now, a database also could archive the application. This led (after a few 
the recipes or at least the part of the iterations) to the software architecture 
recipes that is about temperatures diagram shown in Figure 2. Add to 
and time. Finally, | also mentioned the architecture diagram a Web client 
having a Web page. Data comes from program—for example, Firefox that 
the database, but how do we serve loads and executes the JavaScript code 
the data to the Web page? Because embedded in the Web page. However, 
| also wanted to show the state of the database and Web page are 

the controller on the Web page, | beyond the scope of this article; | may 


Figure 2. Brew 
Controller Software 


Architecture 


BrewController;-> 


Recipe} 
¢ 


PowerController > 


PWMController 
WebSocketSw kK> 


WWW.LINUXJOURNAL.COM / FEBRUARY 2016 / 67 


Temperature Control in a Homebrewing Tun Using a BeagleBone Black 


cover them in a future article. 

| decided to develop in C++, but 
| could have used any programming 
language. In fact, | did start 
developing a prototype in BoneScript, 
a dialect of JavaScript that builds 
on Node.js, which is programmed tn 
JavaScript. It worked quite well, but 
| was unsure whether | could trust it 
to be stable in the long run. | noticed 
during my tests that the BoneScript 
program had some delays a few 
arbitrary times and hiccups | couldn't 
explain. A program, where | know all 
details and that is compiled, is by far 
the most efficient, and once tested 
and debugged, | can trust this one 
to run for long time. And, now | also 
have several different examples of the 
same solution to show my students 
and discuss with them. 


Preparing for Development 

In order to prepare for developing 
the software that fulfills the 
requirements, a few things needed 
to be set up and configured: 

™ Cross compilation. 

m™ Remote debugging. 

® Setting up a temperature sensor. 


@ Setting up the PWM output. 
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@ A log utility. 
® Threading. 


Cross Compilation: | prefer to 
develop using Eclipse when it is a 
larger project that runs over some 
time. There is no problem when 
developing small and quick solutions 
directly on the BeagleBone, but when 
complexity increases, the help that an 
integrated development environment 
provides levels out some of the 
complexity. Can you install Eclipse 
on the BeagleBone? Yes, it should be 
possible, but since the storage on the 
BeagleBone Is Flash-based, | prefer 
to run Eclipse on my laptop. A lot of 
writing to files can wear out the Flash 
memory on the BeagleBone. 

The BeagleBone is built using an 
ARM CPU architecture, and most 
desktops and laptops are Intel CPU 
architectures. This means we need 
to install a cross compiler on the 
laptop. A cross compiler is a compiler 
that produces executable code for 
another CPU architecture than what it 
executes on. In this case, the compiler 
executes on an Intel CPU but produces 
code for an ARM CPU. 

| run Fedora Linux and have been 
doing so ever since the Fedora Project 
was launched. | couldn't find a suitable 
cross compiler in the repositories 


for Fedora, however. So, after some 
searching, eventually | found that 
Linaro.org maintains a suitable compiler 
suite that | could install. If you're 
following along, you should download 
a stable release from the repository. 
This is the one | downloaded: 
http://releases.linaro.org/14.11/ 
components/toolchain/binaries/ 
arm-linux-gnueabihf/gcc-linaro-4.9- 
2014.11-x86_64_arm-linux-gnueabihf. 
tar.xz. There might be fresher copies 
by the time you read this. Take a look 
at http://releases.linaro.org/14.11/ 
components/toolchain/binaries, 
and be sure to download a gnueabiht 
version—the compiler that produces 
code for a hard floating-point unit. 

| keep my downloaded tools, like 
compilers and Eclipses, in the /opt 
directory, so | unpacked the tarball in 
a directory | named toolchains. In this 
directory, you can make a symbolic 
link to the very long directory name 
you will get out of the tarball. This 
makes it easier when configuring 
a project in Eclipse. | made a 
symbolic link called gnueabihf, 
which is far easier to remember than 
gcc-linaro-4.9-2014.11-x86_64_arm- 
linux-gnueabihf. And if | update 
the compiler at a later time, | can 
just remove the link and create a 
new one pointing to the newer 
compiler. | don’t have to change 


anything in Eclipse. 

In Eclipse, you have to select 
the Cross GCC choice in the 
toolchain section of the New Project 
dialog. In the next dialog, enter 
“arm-linux-gnueabihf-” in the Cross 
Compiler Prefix entry. If you look 
into the directory where you stored 
the cross compiler, you will find 
that a/most all binaries are prefixed 
with this. So the gcc is really 
arm-linux-gnueabihtf-gcc. In the 
cross compiler path, enter the 
path to where you installed the 
cross compiler—for example, 
/opt/toolchains/gnueabihf/bin. 

Now Eclipse knows how to produce 
executables that will run on the BBB. 
Remote Debugging: Regardless 

of how experienced we are, bugs 
tend to creep into our code making 
programs not work as expected. So 
the ability to debug is mandatory— 
well, for me at least, but | also do 
have only 30+ years of programming 
experience. Luckily, it is possible to 
set up Eclipse so you can execute the 
freshly compiled program over on the 
BeagleBone under debugger control. 
Included with the gnueabiht 
package is a gdbserver that can 
cooperate with Eclipse or, more 
precisely, with gdb the debugger. 
The gdbserver that comes with 
the BeagleBone by default does 
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not cooperate well with Eclipse. 
So, you have to copy the gdbserver 
to the BeagleBone: 


scp /opt/gnueabihf/bin/gdbserver <username>@<your BBB IP>:~/ 


If you just use the USB-created 
network interface, the address is 
192.168.7.2. | have set up a Wi-Fi 
dongle on my BBB, so my address 
is different. 

The next task is to configure 
Eclipse to perform remote 
debugging. Because this is a rather 
long procedure, | have collected it on 
a page at http://klaus.ede.hih.au.dk/ 
index.php/BBB_Remote_Debugging. | 
made it for my students to follow, so 
readers of Linux Journal also should 
be able to follow these instructions. 


Measuring and Controlling 

Now we’re ready to start 
programming. My typical practise, 
when attacking something | don’t 
know anything or at least not much 
about, is to create small projects 
where | isolate the particular 

problem | am dealing with. One of 
the first things was measuring the 
temperature. | had a DS18B20 1-Wire 
temperature sensor. The sensor is 
embedded in a metal cap, so it is fine 
in food production environments. 
You easily can find them on eBay or 
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similar sites. 

Getting the Temperature: I've 
never worked with 1-Wire devices 
(I just knew they existed), so | had to 
do some reading to understand the 
device, but it is rather simple. Each 
device on the 1-Wire bus has a unique 
address. Now for the purpose of this 
article and the project, you don’t need 
to have a full understanding of the 
1-Wire devices, because the Debian 
Linux that comes with the BeagleBone 
has drivers for 1-Wire devices. So, 
there’s no need to develop a device 
driver ourselves. 

The BeagleBone uses the Flattened 
Device Tree (FDT), like many modern 
Linux distros for embedded systems 
running on an ARM architecture. The 
ARM CPU that is on the BeagleBone 
has an electronic pin multiplexer (a 
kind of switchboard) built in, enabling 
you to connect pins from the outside 
world to a specific internal device— 
for example, GPIO or PWM. The FDT 
will help you set this up, and it tells 
the kernel which type of interface it 
is enabling, so the kernel can use a 
Suitable driver to service the pin. For 
more information on the FDT, see 
http://elinux.org/Device_Tree. 

Basically, the FDT works by having 
a text file with the specification 
of the connection you want to 
enable. This specification file is 


compiled into binary format. You 
may be interested in a generator for 
DTS files, because it takes a lot of 
research and thought to put together 
a DTS. | stumbled upon this site: 
http://kilobaser.com/blog/2014-07-28- 
beaglebone-black-devicetreeoverlay- 
generator#dtogenerator, which 
| find helpful in cooperation with 
the technical specification of 
the BeagleBone (which is at 
https://github.com/CircuitCo/ 
BeagleBone-Black/blob/master/ 
BBB_SRM.pdf?raw=true). 

By looking in the technical 
specification in table 13, | could 
see that pin 12 on the P9 header 
could be connected to a GPIO pin. 
After some testing, | managed to 
put together a device tree file that 
enabled me to connect the DS18B20 
to: PS pin: 1:2: 


/dts-v1i/; 


/plugin/; 


/{ 
compatible = "ti,beaglebone", "ti, beaglebone-black"; 
part-number = "DS1820"; 


version = "Q0A0"; 
exclusive-use = "P9.12"; 


fragment@o { 


target = <&am33xx_pinmux>; 


__overlay__ { 


ds1820_pins: pinmux_ds1820_pins { 


pinctrl-single,pins = <Ox78 0x37>; 


fragment@1 { 
target = <&ocp>; 
_overlay__ { 


onewire@O { 


status = "okay"; 
compatible = "wi-gpio"; 
pinctrl-names = "default"; 
pinctr1-0 = <&ds1820_pins>; 
gpios = <&gpio2 28 O>; 
}; 
}; 
t; 


Save the specification in a file 
called DS18B20-00A0.dts, and 
compile it into the binary format 
using this command: 


root@beaglebone:~# dtc -0O dtb -o /lib/firmware/DS1820-00A0.dtbo 


»-b @ -@ DS1820-00A0.dts 


dtc is a Device Tree Compiler. In 
the above command, It’s instructed to 
output in the dtb format (-0) into a 
file in the /lib/firmware directory (-o). 
The -b sets the boot CPU (here 0) and 
the -@ means to use symbols, and 
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finally, the input file is specified. 
To enable this part of the device 
tree, issue this command: 


root@beaglebone:~# echo DS1820 > 
w/sys/devices/bone_capemgr.*/slots 


If you don’t get any errors, run 1s 
-la in the /sys/bus/w1/devices directory: 


root@beaglebone:~# 1s -la /sys/bus/wi/devices 
total 0 

drwxr-xr-x 2 root root @ Jan 1 2000. 
drwxr-xr-x 4 root root @ Jan 1 2000 .. 

lrwxrwxrwx 1 root root 0 Aug 30 17:32 28-000005a7ce64 -> 
=». ./../../devices/wi_bus_master1/28-000005a7ce64 
lrwxrwxrwx 1 root root © Aug 30 17:32 wi_bus_master1 -> 


=. ./../../devices/wi_bus_master1 


root@beaglebone: /1ib/firmware# 


The 28-000005a7ce64 file (a 
symbolic link) is the thermometer. 
Each 1-Wire device has a unique ID, 
so it will be identified by a pattern 
like 28-O0000nnnnnnn, where 
nnnnnnn is the unique address. 

A quick test to see if the 
DS18B20 is working correctly can 
be done by changing the device 
address in this Python script to 
suit your configuration: 


import time 
wi="/sys/bus/wi1/devices/28 -000005a7ce64/wi_slave" 
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while True: 

raw = open(wi, "r").read() 

print "Temperature is "+str(float(raw.split("t=") 
»[-1])/1000)+" degrees" 


time.sleep(1) 


If you want to have the 
DS18B20 enabled at every boot, 
prepare a file in /etc/init.d, and 
call it enable-DS18B20 or something 
similar. Put this into the file: 


#! /bin/sh 


### BEGIN INIT INFO 
# Provides: enable-DS18B20 
# Required-Start: $all 
# Required-Stop: $all 
# Default-Start: 2345 
# Default-Stop: 0 16 
# Short-Description: Enables the DS18B20 1-wire 
on P9 pin 12 
# Description: Connecting the 1-wire driver to 
the PQ pin 12 
### END INIT INFO 
case "$1" in 

start) 

echo "Enabling 1-Wire DS18B20 on P9 Pin 12" 


echo DS1820 > /sys/devices/bone_capemgr .9/slots 


#no-op 
esac 


exit 0 


After creating the file, execute: 


root@beaglebone: /etc/init.d# chmod 755 enable-DS18B20 
root@beaglebone: /etc/init.d# update-rc.d enable-DS18B20 


defaults 


Try to reboot your system and check 
that the device is enabled as expected. 
Setting Up the PWM Controller: 
As | mentioned earlier, a way of 
controlling the amount of power 


ws ” WaveF orms 
Workspace Settings Window Help 

Welcome = PB Scope 1 £3) | Help 
File Control View Window 

| Single Channel 1 


> scan Mode Auto v | Source 


C2 8000 samples at 800 Hz | 2015-08-03 10:48:15 .430 


4.5 


4 


| 
AL 


Trigger PC 


Figure 3. PWM Signal Output 


¥ | Condition: 


supplied to the heater is to use a 
PWM signal to control a relay. If 
you turn it on for one-third of the 
time and off for two-thirds, the 
applied heat is only one-third of the 
usual amount of heat when running 
switched on all the time. This can 
prevent having particles in the brew 
burn onto the heater. See Figure 3 
for an example PWM signal output. 
Here | have set the period time to be 
approximately one second and the 
“on” time to be approximately 20% 
of the period time. 

As for the 1-Wire, you also need 
a configuration on the device tree 
in order to gain access to the PWM 


(brewController) Avo (x) 
Rising v_ Level: O5V v 

v) Time 
Position: Os v 
Base 1 s/div v 

Add Channel Ms 
v Channel 1 | 
Offset. 2V v 
Range 0.5 Vidiv v 


Channel 2 


Analog Discovery SN:210244465788 | Status:OK = 43 
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controllers on the board and to 
connect it to a suitable pin. If you 
consult the BeagleBone’s technical 
specification, you will find that 


/* the pin header uses */ 
"P9.14", 
/* the hardware IP uses */ 


"ehrpwm1A"; 


P9,14 connects to a PWM controller. 
So, jump to the on-line device tree 
generator and select P9_14 in the 
“Select Pin” box. Then select “Fast 
Slew” in the “Slew” box, “Pullup” 
in “Pullup/Down”, and finally, in 
MuxMode “Mode6é: ehrowm1A”, and hi 
you will get a device tree file like this: }; 


fragment@o { 
target = <&am33xx_pinmux>; 
—__overlay__ { 
bs_pwm_P9_14 0x16: pinmux_bs_pwm_P9_14 0x16 { 


pinctrl-single,pins = <0x048 0x16>; 


* Copyright (C) 2013 CircuitCo fragment@1 { 


* 


Copyright (C) 2013 Texas Instruments target = <&ocp>; 
= overlay... { 


* This program is free software; you can redistribute bs_pwm_test_P9_14 { 


* it and/or modify it under the terms of the GNU compatible = "pwm_test"; 
* General Public License version 2 as pwms = <&ehrpwm1 © 1000000000 1>; 
* published by the Free Software Foundation. pwm-names = "PWM_P9_14"; 
* 
* This is a template-generated file from BoneScript pinctrl-names = "default"; 
a pinctr1-0 = <&bs_pwm_P9_14 0x16>; 
/dts-v1/; 
/plugin/; enabled Sse 
duty =.<05% 
Va i status = "okay"; 
compatible = "ti,beaglebone", "ti, beaglebone-black"; 3; 
3; 
/* identification */ ‘i 
part-number = "BS_PWM_P9_14 0x16"; 3; 


If you want the polarity initially set 
to 0, change the 1 to O in the pwms 


/* state the resources this cape uses */ 


exclusive-use = 
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line so it looks like this: 


pwms = <&ehrpwmi © 500000 0>; 


Again, you need to compile it from 
text format to the binary format 
readable by the kernel: 


root@beaglebone:~# dtc -O dtb -o 
=»/1lib/firmware/bspwm_P9_14 16-00A0.dtbo 
=»-b @ -@ bspwm_P9_14 16-00A0.dts 


To check that you can create a PWM 
device file, execute these commands: 


root@beaglebone~# echo "am33xx_pwm" > 
= /sys/devices/bone_capemgr .9/slots 
root@beaglebone~# echo bspwm_P9_14 16 > 
w/sys/devices/bone_capemgr .9/slots 


If no errors are shown, take a look 
in the device directory: 


root@beaglebone:~# 1s -al 
=»/sys/devices/ocp.3/bs_pwm_test_P9_14.16/ 
total 0 

drwxr-xr-x 3 root root © Jan 1 2000. 
drwxr-xr-x 42 root root © Jan 1 2000... 
lrwxrwxrwx 1 root root © Sep 19 11:40 driver -> 


=»>../../../bus/platform/drivers/pwm_test 


B 


-rW-rw-rw- root root 4096 Mar 1 2015 duty 
-r--r--r-- 1 root root 4096 Sep 19 11:40 modalias 
-rw-rw-rw- 1 root root 4096 Mar 1 2015 period 
-rw-rw-rw- 1 root root 4096 Mar 1 2015 polarity 


drwxr-xr-x 2 root root © Sep 19 11:40 power 


-rw------- 1 root root 4096 Sep 19 11:40 run 
Irwxrwxrwx 1 root root Q@ Jan 1 2000 subsystem -> 
=../../../bus/platform 

-rw-r--r-- 1 root root 4096 Jan 1 2000 uevent 

The files of interest are “polarity”, 
“period” and “duty”. The polarity 
controls whether the controller 
outputs O or 1 when turned off. The 
period, as the name indicates, controls 
the period time—that is, for what 
length of time is a complete cycle in 
the controller. The period time is set in 
nanoseconds. Finally, the duty controls 
the amount of time that the signal 
from the PWM controller is active. 

The duty is set in percentages—for 
example, for 50%, you would write 
0.5 to the duty file. 

And as for the temperature sensor, 
an automatic start at boot would be 
handy, so create this file in /etc/init.d/ 
enable-pwm with this content: 


#! /bin/sh 


### BEGIN INIT INFO 

# Provides: enable-pwm 

# Required-Start: $all 

# Required-Stop: $all 

# Default-Start: 2345 

# Default-Stop: 0 1 6 

# Short-Description: Enables the PWM chips and 
# connects it through the pinmux 


# Description: Connecting the pwm output through 
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# the pinmux and enables the PWM chip on board 

### END INIT INFO 

case "$1" in 

start) 
echo "Enabling PWM on P9 Pin 14" 
grep -q am33xx_pwm /sys/devices/bone_capemgr.9/slots | | 
echo "am33xx_pwm" \ 
> /sys/devices/bone_capemgr .9/slots 


echo bspwm_P9_14 16 > /sys/devices/bone_capemgr .9/slots 


#no-op 


#no-op 


exit 0 


Reboot your BeagleBone and check 
that you get the PWM device files 
created as shown above. 

Now you're ready to develop 
a program that will measure the 
temperature, and based on a recipe, 
calculate the necessary power to apply 
to the heater. 

A Simple Log Utility: When 
developing a program, it’s often 
necessary to log different values from 
the program, especially when you 
have several threads running. Nothing 
happens at the time you would expect 
it to—this is the nature of threading. 
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Therefore, | often print out to the 
console or log to a file, which lets me 
follow the progress of the program. 

In this project, | started printing 
to the console but switched to 
logging to a logfile. | searched the 
Internet and ran across this site: 
http://www.infernodevelopment. 
com/c-log-file-class-forget-debuggers. 

The header looks like this: 


#include <fstream> 
using namespace std; 


class Log 
i 
public: 
Log (const char* filename); 
~Log ( ); 
void Write (const char* logline, ...); 
private: 


ofstream m_stream; 


ti 
And the implementation is simple: 


#include "Log.h" 


#include <stdarg.h> 


Log::Log (const char* filename) 


if 


m_stream.open (filename); 


Log::~Log ( ) 
{ 


m_stream.close (); 


void Log::Write (const char* logline, ...) 
{ 
va_list argList; 
char cbuffer[1024]; 
va_start(argList, logline); 
vsnprintf (cbuffer, 1024, logline, argList); 
va_end(argList); 


m_stream << cbuffer << endl; 


This version allows me to log 
using the sprintf conversion 
specifiers—for example, %f for 
floats. | use it like this: 


log->Write("PowerController: INFUSION POWER: \ 

PWM set to %F\n", maxPWM) ; 

Notice here, that | use %f and pass 
a float value, which will be logged 
just like (s)printf would convert the 
float to a string of digits. This is due 
to the varadic declaration of the 
Write function. 

When you want to see the log 
while the program runs, open 
another SSH connection to the 
BeagleBone and run: 


root@beaglebone:~# tail -f <logfilename> 


where you, of course, need to 
change <logfilename> to match 
your current log file. 

Threading: | know that C++ ISO 
standard 2011 (or just C++11) allows 
you to create threads in C++ directly, 
but prior to that, | needed a way to 
make classes (or a function in a class) 
into a thread. 

| found an example class on 
StackOverflow that | have been using 
ever since. So why learn something 
new, when you know how to do it? 
We are all a bit lazy now and then, 
aren't we? Here it Is: 


#include <pthread.h> 


#include <cstdlib> 


class Threadable 
{ 
public: 
Threadable() 
{ 
_thread = (pthread_t)NULL; 
} 
virtual ~Threadable() 
{/* empty */ 


} 


/** Returns true if the thread was successfully 
started, false if there was an error starting 
the thread */ 

bool StartInternalThread() 


{ 
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return (pthread_create(&_thread, NULL, 


»InternalThreadEntryFunc, this) == 0); 


/** Will not return until the internal thread has exited. */ 
void WaitForInternalThreadToExit( ) 
ae 

(void) pthread_join(_thread, NULL); 


} 


pthread_t GetThreadID () { return _thread; } 


protected: 
/** Implement this method in your subclass with 
the code you want your thread to run. */ 


virtual void InternalThreadEntry() = 0; 


private: 
static void * InternalThreadEntryFunc(void * This) 
{ 
((Threadable *) This)->InternalThreadEntry(); 


return NULL; 


pthread_t _thread; 


This is a virtual base class that 
you can’t create instances of 
directly, but you will inherit from 
it in a new sub-class. 

So, for instance, my BrewController 
class begins like this: 


class BrewController: public Threadable 


{ 
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public: 
BrewController ( ); 
virtual ~BrewController ( ); 
void InternalThreadEntry ( ); 


The InternalThreadEntry (_) is 
the thread function for this class. So 
you will just have to fill in the code 
that composes the thread. The rest of 
the methods in the class are helper 
functions in one way or another. 


Controlling the Heater 
In order to control the heater, we 
need to measure the temperature in 
the mash. Having a representation 
of the temperature, it is a matter of 
comparing the current temperature 
with the desired temperature for the 
current step in the recipe. We use a 
PID controller to calculate the amount 
of power needed, but due to our 
experience with particles in the mash 
burning onto the heater, it may be 
necessary to limit the power. So the 
PID controller may call for full power, 
but the applied power is limited. 
Measuring the Temperature: 
The first thing to do in the program 
is to locate the thermometer or the 
temperature sensor. | want the program 
to locate any DS18B20 attached to 
the BeagleBone. As mentioned earlier, 
the 1-Wire devices all have unique 


addresses. | currently have five or six 
DS18B20 thermometers on hand, so 
| designed the software so that | can 
attach any of my thermometers to 
the BeagleBone, and the program will 
locate it and start using It. 

This code snippet does the trick 
(it is from a class | call Thermometer): 


int Thermometer: :locateThermometer ( ) 

{ 
string initialDir = "/sys/bus/wi/devices/"; 
string regExpr = "28-00000"; 


string dir; 


// Open directory 
DIR *dp = opendir(initialDir.c_str()); 
if (!dp) 
{ 
exit (EXIT_FAILURE); 


} 


struct dirent *dirp; 


// Loop through the directory entries 
while ((dirp = readdir(dp)) != NULL) 
{ 
std::size_t found = \ 
string(dirp->d_name) .find(regExpr) ; 
if (found != std::string: :npos) 
< 
// We found one entry that matches 
oneWireDir = initialDir + string(dirp->d_name)\ 
+ string("/wi_slave"); 
// Nicely close the directory again 


(void) closedir(dp); 


return 0; 


} 


exit (EXIT_FAILURE) ; 


oneWireDir is a class variable where 
| keep the directory and file from which 
| can read the temperature sensor. 

The output from a DS18B20 is made 
up of different information. Take a 
look at the output below: 


root@beaglebone: /sys/bus/w1i/devices/28-000005a7ce64# cat wi_slave 
85 01 4b 46 7f ff Ob 10 5f : crc=5f YES 
85 01 4b 46 7f ff Ob 10 5f t=24312 


root@beaglebone: /sys/bus/w1/devices/28 -000005a7ce64# 


The first line is of no interest to 
us. The second line shows that the 
temperature t is 24.312°C. From this, 
we learn that it is just a matter of 
reading the w1_slave file and grabbing 
the temperature from the output. | have 
designed this function to do the task: 


int Thermometer: :readTemperature( ) 


{ 
aint fd: 


int res; 


// Open the OneWire thermometer file 

fd = open(oneWireDir.c_str(), O_RDONLY); 
if (!fd) 

‘ 
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return -1; 
} 
char buf[256]; 
// Read from it 
res = read(fd, buf, sizeof(buf)); 
if (res < 0) 
{ 
close (fd); 
return -1,; 
} 
// Retrieve current temperature from device 
if (res > 0) 
4. 
std::size_t found = string(buf).rfind('='); 
if (found != std::string: :npos) 
{ 
curTemp = atof((const char *) \ 


&buf[found+1]) / 1000; 
else 


close(fd); 


return -1; 


} 
close (fd); 
// Call calc average temperature 


return calcAvgTemp(); 


The file is opened and read, and 
because there are two = characters in 
the buffer, | perform a reverse search 
(from the end of the buffer rather 
than from the beginning) in the 
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buffer for =. When the = is located, | 
convert the ASCII text to a float using 
atof and scale it with 1,000 since 
there are no decimal points in the 
readout from the sensor. 

The next thing to do is to calculate 
the average temperature. | keep the 
last five samples in an array in order 
to calculate an average temperature. 
The reason for this is that | occasionally 
Saw some spurious measurements 
where the temperature was a bit off. 
The averaging will smooth this out: 


int Thermometer: :calcAvgTemp() 
{ 
float t = avgTemp - curTemp; 
if ((t < -20 || t > 20) && tempSamples.size() > 4) 
{ 
// Skip this measurement - it's way out of range. 
log->Write("Skipped this temperature: %f, \ 
t = %d\n", curTemp, t); 
return -1; 
i 
if (tempSamples.size() > maxSamples) 
{ 
// Get rid of first element 
tempSamples.erase(tempSamples.begin()); 
} 
tempSamples.push_back(curTemp) ; 
t = 0; 
for (int i = 0; i < (int)tempSamples.size(); i++) 
{ 


t += tempSamples[i]; 


u(t) 


Plant / 
Process 


Figure 4. PID en updated feedback by TravTigerEE—Own work. Licensed under CC BY-SA 3.0 


via Commons. 


avgTemp = t / tempSamples.size(); 


return 0; 


maxSamples currently is set to 5. As 
you can see, if the vector that holds the 
samples is filled, | drop measurements 
that are 20°C off the average. If the 
vector is not filled, | add the sample to 
be included—it is needed when starting 
the measurements. 


From this point, we just need to set 
up, somewhere else, a timed job that 
calls the readTemperature() function. 
| have set it up to read every tenth 
second. With a mass of approximately 
30kg, it is a relatively slow process. 

A PID Controller: What is a PID 
controller? It is a proportional- 
integral-derivative controller. From 
that, you can see that it is a controller 
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that reacts proportionally on the 
measured input compared to the 

set point (the desired value), and it 
accumulates the historic development 
of the controlled using an integral 
function, and finally, it tries to predict 
the future in the derivative part. That 
was a big mouthful. Let’s take this one 
step at a time. 

See the diagram of a PID controller 
in Figure 4 or expressed as a formula in 
Figure 5. 

Our “plant” (as shown in Figure 4) 
is the brew tun. We measure, like 
in this project, a temperature, 
and elsewhere, we have a desired 
temperature we want to keep for a 
while. The difference between the 
measured and desired temperature Is 
called the error—e(t) in the Figure. 

The error signal is fed into the 
proportional, integral and derivative 
part of the controller. 

The Proportional Controller: If the 
measured temperature Is lower than 
the desired, we can fire up the heater 
to raise the temperature. If it is higher, 
we can do nothing in this setup but 
wait for it to cool down a little. 

The amount of power that we'll 


de 


t 
u(t) = K,e(t) + K; I e(r)dr + Ka 


Figure 5. PID Formula 
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apply to the heater is controlled 

by the error signal. If the error is 
small, a small amount of power is 
applied, and if it is large, we apply 
more power to the heater. By having 
a factor (Kp) to multiply with the 
error, we can amplify the error, or if 
the factor is below 1, attenuate the 
impact of the error. The proportional 
part of the PID controller tends to 
overreact—that is, the temperature 
overshoots the desired temperature 
by quite an amount. This Is not 
desired in a process where we are 
seeking to give our enzymes ideal 
temperatures to work in. And if 
enzymes get too warm, they will 
denature—that is, die! The derivative 
part will compensate for this to 
some degree. 

The Integral Controller: The 
integral part of the controller sums or 
accumulates the error over time. This 
part can suppress the proportional 
part, but with the cost of being slower 
to reach the desired set point. 

The Derivative Controller: This 
part looks into the future, so to 
speak. It compares the previous 
measurement with the current one 


in order to predict the future state 
of the plant. This results in a quicker 
raise to the desired set point, but it 
also will back off when nearing the 
set point, leaving the fine-tuning to 
the P and | parts. 

For further details, take a look 
at this fine article by Wess Scott: 
http://m.eet.com/media/1112634/ 
f-wescot.pdf, or see this page on 
Wikipedia: https://en.wikipedia.org/ 
wiki/PID_controller. 

The PID Code: It’s rather easy to 


implement a PID controller in software: 


// The PID Controller 


float processVal = curTemp; 


// The Proportional part 
float error = setPoint - processVal; 


float pTerm = pGain * error; 


// The Integral part 
iState = iState + error; 


float iTerm = iState * iGain; 


// The Derivative part 
float dTerm = (dState - processVal) * dGain; 


dState = processVal; 


// Scaling to fit 


float pwr = (pTerm + iTerm + dTerm) / 10; 


The factors (Kp, Ki and Kd) that | 
currently use are set to: 


pGain = 5.0; // Kp 
iGain = 0.025; // Ki 
dGain = 30.0; // Kd 


| let the proportional and derivative 
parts have quite a large impact, and the 
integral part is dampened quite a bit. 

So after getting the PID controller 
set and working as desired (although 
some fine-tuning can be done on the 
Kn parameters in order to find the 
optimal for my set up), it’s time to 
turn to the PowerController. 

Controlling the Heater: The output 
from the Thermometer class is fed 
into the PowerController. This class 
uses the PID controller to calculate the 
amount of power to apply. 

As mentioned before, | had to limit 
the amount of power applied even 
if the PID controller called for more. 
The reason Is that below 57°C-59°C, 
we have determined that if we apply 
one-third of the available power, we 
will not cause anything to burn onto 
the heater. Therefore, | have set some 
limits, regardless of how much power 
the PID controller calls for. 

But hey, what about infusion? Infusion 
is the point in time where we add the 
malt to the preheated water. It’s also 
called the strike temperature. The colder 
malt mixes with the water and cools it a 
little. Therefore, the infusion temperature 
can be calculated from the mass of water 
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T ivike —_ 


fen? (a + (0.4 * Wwait)) — (0.4 * Wyatt * Tuait) 


Vwater 


Tstrike : Strike temperature in Celsius [°C] 
Tash: desired mash temperature in Celsius [°C] 


Tat: The temperature of the malt in Celsius [°C] 


Vwater: volume of water in litres [L] 
Wnat : Weight of malt in kilos [kg] 


Figure 6. Strike Temperature Calculation 


and malt using the amount of them 
both and the temperature of the malt 
(Figure 6). So if the desired infusion (or 
strike) temperature is 55°C, the amount 
of water is 19 liters, and the 7kg of 
malt is 14°C, we can calculate that the 
water should be preheated to 59.8°C 
before adding the malt. After a short 
while, it’ll find its rest at 55°C. 

In order to get ready for brewing as 
quickly as possible, we can heat the 
water with full power until we add the 
malt. So we need to know the different 
steps of the mashing process, which 
is kept in a database (more about that 
later). From that, we get the set point. 

| have put the maximum 
temperature ranges into a 
two-dimensional array of floats: 
tempPwrRange[0][0] = 57; // Below this temp. use 


tempPwrRange[0][1] = 0.333; // this amount of pwr 
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tempPwrRange[1][0] = 60; // Below this temp. use 


tempPwrRange[1][1] = 0.666; // this amount of pwr 
tempPwrRange[2][0] = 102; // Below this temp. use 
tempPwrRange[2][1] = 1.0; // this amount of pwr 
Then, it’s simple to compare 
against the current temperature and 
find the maximum amount of power 


that may be applied: 


// Locate the temperature range to operate within 
// Use the current temperature measured to retrieve 
// the maxPwr 
for (int i = 0; i < 3; it+) 
{ 

limitTemp = tempPwrRange[i][0]; 

maxPwr = tempPwrRange[i] [1]; 

if (curTemp > limitTemp) 

{ 


continue; 


After having the PID controller 
calculate the desired power, it is left 
to the setPwrLvl1 function to find 
what can be set according to the 
current step in the recipe from where 
we have derived the maxPwr: 


float PowerController::setPwrLvl (float pwr, 


float maxPwr) 


// The PWM operates with a high granularity, 


// hence we have to scale the pwr 


// Corrects the duty cycle when applied to the PwM 
const long long Factor = 999000000; 
log->write("PowerController: maxPwr is set to %f",\ 


maxPwr ) ; 


// If the requested pwr is less that 0 
// we set it to 0 
if (pwr < 0) 
{ 
// Set PwM to 0 
if (pwm.setDutyCycle (0) == 0) 
t. 


log->write("PowerController: PWM set to 0\n"); 


else 


log->write("PowerController: ERROR: \ 
PWM not set! (pwr<@)\n" ); 


Z 


return 0.0; 


// If we are in the infusion step it is allowed to use max pwr 


if (infusion) 


{ 
if (pwm.setDutyCycle (maxPWM) == 0) 
{ 
log->write("PowerController: INFUSION POWER: \ 
PwM set to %Ff\n", maxPWM) ; 
} 
else 
{ 
log->wWrite("PowerController: ERROR: PWM not\ 
set! value: %f\n", maxPWM) ; 
5 
return 1.0; 
} 


// If the pwr is less than the maxPwr allowed 
// use the requested pwr 


if (pwr < maxPwr) 


{ 
// Set PWM to pwr 
if (pwm.setDutyCycle (pwr * Factor) == 0) 
{ 
log->write("PowerController: PwM set to \ 
%f\n", pwr * Factor); 
} 
else 
{ 
log->write("PowerController: ERROR: PWM not \ set!\n"); 
} 
return pwr; 
} 


// Otherwise use the maximum power, i.e. 100% 


else 
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// Set PwM to the maxPwr 
if (pwm.setDutyCycle (maxPwr * Factor) == 0) 
{ 

log->write("PowerController: PWM set to \ 


%f\b", maxPwr * Factor); 


else 


log->write("PowerController: ERROR: PWM not \ 
set! \n"); 
3 
return maxPwr ; 


af 


return 0.0; 


return -1; 


// Write the requested duty cycle to the file 
char str[100]; 

sprintf(str, "%lld", duty); 

size_t res = write (f, str, strlen(str)); 


close (f); 


// Handle errors 
if (res != strlen (str)) 
{ 
log->Write("ERROR Opening file %s with this \ 
error: %s\n", 
dir c.str(); 


strerror( errno )); 


return -1; 
The setDutyCycle function actually } 
writes to the PWM file that represents return 0; 
the PWM controller on board: } 
int PwMController::setDutyCycle(long long duty) Yes, | know—l am a C hacker (mis-) 
{ using the fine C++ language with file 
string dir; operations in plain C code. But, so it 
is! What comes easiest to the fingers 
dir = initialDir + "/duty"; over the keyboard is what goes In the 


code—sometimes | think there may 
// Open the file representing the PwM controller not be a brain involved at all. But 


int f = open(dir.c_str(), O_WRONLY); otherwise, it’s straightforward code. 
if (f < 0) This concludes the measuring of 
i the temperature and controlling the 
log->Write("ERROR Opening file %s with this \ power output. In a future article, "I 
error: %s\n", look into adding an SQLite database 
dir.clstr(); to the brewController program. In 
strerror( errno )); order to service a Web page, we also 
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need a WebSocket server, so | will 
introduce that as well. 

The database has two purposes: 
storing recipe details for the mash 
process and continuous logging of 
data produced during a brew. 

The WebSocket server is the 
interface to the Web page from 
where we can monitor and control 
the brewing process. So | also will 
dive into a relatively complex Web 
page with a lot of JavaScript code 
to control the graphs and retrieval 
of data from the WebSocket server, 
which will look up in the database for 
the correct data to serve. 

Finally, I'll write about how to 
dzemonise the brewController and 


Resources 


make it start automatically when the 
BeagleBone Is powered. 

All the code will be open-sourced 
when it has proven to control a few 
brews this autumn. If you are ina 
hurry, send me an e-mail and | will 
send you the code.m 


Klaus Kolle is currently teaching electronic engineering 
students software development and device driver development 
in the Linux kernel at Aarhus University. Klaus has been 
working and developing on UNIX and Linux since 1988. 

Klaus loves well brewed beer, especially if it is home-brewed. 


TEE 
Send comments or feedback via 
http://www.linuxjournal.com/contact 
or to ljeditor@linuxjournal.com. 


You can retrieve a cross compiler for the ARM processor at 
http://releases.linaro.org/14.11/components/toolchain/binaries/arm-linux-gnueabihf/ 
gcc-linaro-4.9-2014.11-x86_64_arm-linux-gnueabihf.tar.xz. 


My instructions to set up the remote debugging in Eclipse: 
http://klaus.ede.hih.au.dk/index.php/BBB_Remote_Debugging 


About the Flattened Device Tree (FDT): http://elinux.org/Device_Tree 


A Web page for setting up the Device Tree specification files: http://kilobaser.com/ 
blog/2014-07-28-beaglebone-black-devicetreeoverlay-generator#dtogenerator 


A simple log class: http://www.infernodevelopment.com/c-log-file-class-forget-debuggers 


The technical specifications for the BeagleBone Black: 
https://github.com/CircuitCo/BeagleBone-Black/blob/master/BBB_SRM.pdf?raw=true 


PID without a PhD: http://m.eet.com/media/1 112634/f-wescot.pdf 


Wikipedia on PID controllers: https://en.wikipedia.org/wiki/PID_controller 
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Does Every 


Year Havea 
Friday the 13th? 


Can you write a one-liner that answers this question for 
a given year? Can you write a short script around your 
one-liner and solve this puzzle for all time? 

SOL LEDERMAN 
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ere’s a fun little math problem 

that a handful of Linux 

commands, a bit of thought 
and a short shell script will help solve: 
does every year have a Friday the 
13th? cal, cut, grep, sed and cksum 
will be your friends in this exploration. 

Before engaging the computer, 
let’s consider how you might solve 
this problem without one; you could 
examine some number of calendars. 
If you discover a year’s calendar with 
no Friday the 13th, you’re done. 
Otherwise, you eventually will come 
to believe that every year does have 
a Friday the 13th. “Eventually” is a 
long time away though. So, how many 
calendars do you need to look at? You 
may want to pause to consider this 
question before reading further. And, 
to support your exploration, you may 
want to use the Linux cal command 
to display calendars to your terminal. 
Welcome back! Did you count 14 

different calendars that you need to 
consider? Did you count seven? Both 
answers are correct, depending on 
your approach. (If you don’t believe 
that looking at seven calendars 
is enough, see the suggested 
explorations at the end of the article.) 
Let's also consider leap years for this 
exploration. And, there’s another 
consideration: is it enough to inspect 
the calendars for 14 consecutive 


years? Ponder that for a bit. 

Let’s dive in. How many unique 
calendars are there? Two calendars are 
“unique” if they look different—that 
is, if Some months start on different 
days when you compare the two. 
Two calendars are the same if you 
could hang either one on the wall, 
and except for the year printed at 
the top of the calendar, you couldn't 
tell the difference. 2014 started on 
a Wednesday, and 2015 started on 
a Thursday. Those are two unique 
calendars. 2006 and 2012 both 
started on a Sunday. Are they the 
same calendar? Could you swap one 
for the other? No, because 2012 was 
a leap year, but 2006 wasn’t. So, 
although January of both calendars 
was the same for those years, the 
other months weren't. 

Considering that some years are leap 
years and that others aren't leads to 
one approach to counting calendars. 
Every year starts on one of the seven 
days. If you don’t consider leap years 
(and there is an approach where you 
don’t need to consider them), then 
there are seven different calendars. If 
you also consider leap years, you have 
seven more calendars to consider. So, if 
you examine 14 unique calendars, you 
will have the answer. You either will 
find a year that doesn’t have a Friday 
the 13th or you won't. 
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Here is the part of the analysis where 
the computer is going to help. You 
need to look at 14 unique calendars. 
Will it be enough to look at the 
calendars for 2000 through 2013? 
That's a 14-year span. It turns out that 
that span does not include 14 unique 
calendars. In particular, 2001 and 2007 
are the same calendar. Rather than 
trying to think through how many 
calendars you need to look at, let’s 
take a different approach. Let's write 


phone line. You want some assurance 
that the file wasn’t corrupted during 
the transmission. What could you 

do? You could compare the sizes of 
the files on both ends. If the sizes 

are different, you would know the 
file copy is corrupt. But, if the sizes 
are the same, you certainly can’t be 
confident that the content of the files 
is the same. You could transmit the 
file twice and compare the two copies 
byte by byte, but that would double 


Checksums solve the problem of 
verifying that two files are identical 
without incurring a large 
computational or transmission cost. 


a shell script that will review calendars 
until it has looked at 14 unique ones. 
How can you make the computer 
tell you if calendars for two years are 

the same? Let's take a detour away 
from that question for a short while 
and consider this command: cksum. 
cksum displays the checksum of a file 
(or of stdin). What's a checksum? A 
checksum is a number associated with 
a file that is commonly used to detect 
transmission errors. 

Imagine that you send a file from 
one computer to another via a noisy 
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the transmission time. 

Checksums solve the problem 
of verifying that two files are 
identical without incurring a large 
computational or transmission cost. 
The Linux cksum utility computes 
a single number—the checksum, 
for a file. The process for using 
the checksum is this: compute the 
checksum, transmit the file, compute 
the checksum for the received file 
and compare the checksums. If the 
checksums are identical, there is a 
near-100% probability that the two 


files are identical. Near-100% is quite 
good enough for most purposes. 

Now, let’s return to the calendar 
problem. You want to find 14 unique 
calendars and check each to see if 
they have a Friday the 13th. Here’s 
some pseudo-code that illustrates 
this approach: 
Year = 2000 # Year increases until we 
# have seen 14 unique 
# calendars 
ChecksumCount = @ # This counts how many unique 
# calendars we've seen 
While ChecksumCount < 14 { 

Compute the checksum for Year 
If we have not seen this checksum yet { 
Add this checksum to the list of 
checksums we've seen 
Add 1 to ChecksumCount 
} 
Add 1 to Year 


Why do you use checksums to 
compare calendars? Why don’t you just 
create a file for each unique calendar 
and compare each calendar to every 
saved file? You could do that. But, 
once you work through the checksum 
approach, | think it will become clear 
that comparing files, with the diff 
command or some other approach, Is 
clunkier. You decide. 

It’s time to dive in to some shell 


commands. | mentioned earlier that 
2001 and 2007 are the same calendar. 
Compare the output of these two 
commands: 


cal 2001 
cal 2007 


Now, compute the checksum of 
each calendar by piping the output of 
cal to cksum: 


cal 2001 | cksum 
3673415557 2014 


cal 2007 | cksum 
2655244645 2014 


Hmmm. The two checksums are 
not the same. Why is that? (Note that 
“2014” is the number of characters 
in the output of cal. Ignore that and 
just look at the first number.) It’s the 
title in the output of cal (the year) 
that makes the two calendars appear 
to be different. All you need to do is 
remove the first line, and you'll be 
good. sed is one of a number of Linux 
tools that easily can remove the first 
line of the output: 


cal 2001 | sed '1d' | cksum 
2408573533 1980 
cal 2007 | sed '1id' | cksum 


2408573533 1980 
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Note that cksum may generate 
a different checksum for your cal 
output. That’s because cal has slightly 
different layouts on some flavors of 
Linux. As long as you're comparing 
cal output on the same machine, 
everything will work correctly. 

cksum has confirmed that 2001 
and 2007 are the same calendar. 
The next task is to build a list of the 
unique checksums that you’ve seen so 
that you can compare new calendar 
checksums to those on the list. Let's 
use a String to hold that list and 
initialize it to be empty: 


checksums="" 


Then, add the checksum for 2001 
to the list: 


checksums="$checksums 2408573533" 


In order to add just the checksum 
to the list and not the number of 
characters in the output of cal, you 
will need to keep just the first part 
of cksum’s output and discard the 
second part. There are many ways to 
do this in bash. Here's one way: 


cal 2001 | sed '1d' | cksum | awk '{print $1}' 


2408573533 


Awk is a very powerful text and 
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data processing Linux tool. If you've 
never used awk, it’s worth reading a 
tutorial about it and exploring it. For 
the purposes of this article, all you're 
doing with awk is telling it to print 
the first (whitespace-separated) field 
of the input ($1) from the pipe. 

With the new checksum, 
2408573533, use grep to search the 
list of checksums you've seen for this 
new one: 


echo "$checksums" | grep -w 2408573533 


Note that it’s a good idea to use 
the -w flag with grep to match only 
2408573533 as a word, with spaces, 
tabs and punctuation separating 
words. You don’t, for example, want 
to match 2408573533123456789. 

You also will want to use grep with 
the -q option when you write your 
Shell script to count Friday the 13ths. 
The option tells grep to search quietly 
and not to display any output. You then 
will use the return status from grep to 
know whether it matched anything: 


echo $checksums | grep $newChecksum 
if [ $? -eq 0 ] # if return value is 0 
then 
# we matched something 
fi 


Whenever you see a unique 


calendar, increment a counter: 


numUniqueCalendars=$( (numUniqueCalendars+1) ) 


$((..)) iS a way you can do simple 
arithmetic in bash. Note that inside 
the $((..)), you don’t put dollar 
signs in front of variable names. 

When the counter reaches 14, 
you've seen all unique calendars. 

Now that you have a general idea 
about how to use checksums, strings 
and grep to determine whether 
a calendar is unique and to count 
unique calendars, there is one more 
significant task. Given a calendar, 
how do you know if it has a Friday the 
13th? Let's look at the output from 
cal for just one month: 


cal 4 2001 
April 2001 

Su Mo Tu We Th Fr Sa 
123 4 5 6 7 
8 91011 12 13 14 
15 16 7 26 19-2021 
22 23 24 25 26 27 28 
29 30 


April 13, 2001, is a Friday. It’s easy 
enough for us humans to scan the 
Friday column looking for 13. How 
might a computer do the same? If 
you're not familiar with the Linux 
cut command, take a look at its 


man page and play with it a little 
before reading further. 

cut allows you to select one or 
more columns. Sunday dates are 
in columns 1 and 2; column 3 is 
blank; Monday dates are in columns 
4 and 5; and, if you keep counting 
columns, you see that Friday dates 
are in columns 16 and 17. Pipe the 
output of cal to cut, and you get 
all the Friday dates: 


cal 4 2001 | cut -c16-17 
Fr 


All you need now is to grep these 
Friday dates for the number 13: 


cal 4 2001 | cut -c16-17 | grep 13 
13 


So, you now have a Linux pipeline 
that will tell you if a given year and 
month have a Friday the 13th. How 
can you generalize this approach to 
scan an entire year? Note that cal 
displays entire years in a 3x4 layout 
of three months across and four 
months down. You can use cut again 
and have it display three columns of 
Fridays. Note that you will need to 
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include a column with a space, either 
before or after the dates, otherwise 
the column values will run together: 


cal 2001 | cut -c16-18, 38-40, 60-62 


Note that the column ranges may be 
different in your version of Linux. | had 
to change column numbers to get the 
output to be correct on one of my Linux 
machines. Here is some of the output: 


Fr Fr Fr 
6 4 1 
13 41 8 
20 18 15 
a7 25 22 

29 


Notice that there are two Friday 
the 13ths in 2001. Let’s add grep to 
the pipeline to filter out just the lines 
with 13 in them and use we to count 
those lines: 


cal 2001 | cut -c16-18,38-40,60-62 | grep -o 13 | we -l 


2 
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You may be wondering why 
you need the -o option to grep. 
Consider this: 


echo 12 13 14 13 | grep 13 
12 13 14 13 


The number 13 appears twice in 
the list that you echo, but grep is 
matching the entire line, and when 
you pipe the output of grep to 
we -1 to count matches, you get this: 


echo 12 13 14 13 | grep 13 | we -l 
Hl 


It would be nice to know how many 
Friday the 13ths there are In a year. 
This is where the -o option is useful: 


echo 12 13 14 13 | grep -o 13 
13 
13 


Adding -o tells grep to print only 
the matching part of lines and to print 
matches on multiple lines. Now you 
easily can count Friday the 13ths for 
any year. How many Friday the 13ths 
were there in 2014? Let’s see: 


cal 2014 | cut -c16-18,38-40,60-62 | grep -o 13 | we -l 


all 


Congratulations! You've laid all of 


the groundwork and should now be 
able to write a shell script to answer 
this article’s burning question! 

Let’s summarize the programming 
approach used here. This should 
look very similar to the pseudo- 
code earlier in this article with one 
addition: you are going to count the 
number of Friday the 13ths in each 
unique calendar. 


1. Pick a starting year. Does it matter 
what that year is? 


2. For this year, compute its 
calendar checksum. 


3. If the checksum is not on the 
unique checksum list, then you 
are looking at a calendar that 
you've not looked at previously. 
Add its checksum to your unique 
checksum list, and add one to 
the number of unique calendars 
you've seen. Count the number 
of times Friday the 13th appears 
in this year. 


4. |If you've seen 14 unique calendars, 
you're done. If not, add one to the 
year and loop back to step 2. 


Here is a bash script that puts 
all of these ideas together. Note 
that the lines ending in \ need to 


not have a space, tab or any other 
character after them. 


#!/bin/bash 
# Starting with the year 2000, we 
# compute how many times 
# Friday the 13th occurs in every 
# year. We stop when we have 
# looked at 14 different calendars. 
year=1999 
checksums="" 
numUniqueCalendars=0 
echo "Year Number of Friday the 13ths" 
while [ $numUniqueCalendars -1lt 14 ] 
do 
year=$((year+1) ) 
sum=$(cal ${year} | sed '1d' | cksum |\ 
awk '{print $1}') 
echo "$checksums" | grep -q $sum 
if [ $? -ne 0 ] 
then # It's a new calendar 
checksums="$checksums $sum" 
numUniqueCalendars=$((numUniqueCalendars+1) ) 
echo -n "${year} " 
cal ${year} | cut -c16-18,38-40,60-62 |\ 
grep -o 13 | we -l 
fi 


done 
Here’s the output: 


Year Number of Friday the 13ths 
2000 si 
2001 2 
2002 2 


WWW.LINUXJOURNAL.COM / FEBRUARY 2016 / 95 


FEATURE Command-Line Tutorial: Does Every Year Have a Friday the 13th? 


2003 
2004 
2005 
2006 
2008 
2009 
2010 
2012 
2016 
2020 
2024 


NN F&F WwW rF WwW PY FD FB 


Notice that you had to examine 
25 consecutive calendars (between 
2000 and 2024) to find 14 unique 
ones. And, notice...drum roll... 
Friday the 13th occurs once, twice 
or three times in every year. 

| chose to start checking calendars 
starting in 2000. What would have 
happened if | had chosen some 
other year? 

| hope you appreciate the power 
of a handful of Linux commands, 
some logical thinking and a short 
shell script to solve a fun math 
challenge. If you enjoyed this 
problem, consider exploring these 
related problems: 


1. How would using ncal instead of 
cal, if it’s available on your system, 
simplify your code? 


2. Does every year have a Monday the 
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13th? What's the most any year can 
have? For this exploration (and for 
#6), you'll need to generalize the 
concept of pulling out one or more 
columns from a calendar to be able 
to select any day of the week. 


3. If you exclude January and 
February, does every year still have 
a Friday the 13th? 


4. 2015 has three Friday the 13ths. 
When will that happen next? 


5. How likely is the 13th of a month 
to fall on a Friday vs. the other days 
of the week? Have the computer 
scan every month for 100 years of 
calendars to find out. 


6. Which day/date pairs occur every 
year? For example, does every 
year have a Monday the 31st? This 
exploration generalizes #2.m 


Sol Lederman is a math geek and general techie. As a 
writer and programmer, he especially loves to communicate 
technical things to a broad audience. Read more about 
Sol at http://sollederman.com. 
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@ macromaker.module % macromakerjs x jquery.insertatCaret js 


3 * 

4 * Usage: $(obj).insertAtCursor(text); 

5 * obj = a textarea or textfield 

6 * text = a string to insert 

7 */ 

8 $.fn.extend({ 

So insertAtCaret: function(myValue){ 

100) return $(this).each(functiond){ 

11 // If target element is hidden, don't do it. 
12 if ($Cthis).isC' :hidden") || $C ).parents(' :hidde 
13 return; t 


if Cdocument.selection) { 
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= ==== Maximizing NoSAQL Clusters for Large Data Sets 


This follow-on webcast to Reuven M. Lerner's well-received and widely acclaimed Geek Guide, "Take Control of Growing Redis 
NoSQL Server Clusters", will extend the discussion and get into the nuts and bolts of optimally maximizing your NoSQL clusters 
working with large data sets. Reuven's deep knowledge of development and NoSQL clusters will combine with Brad Brech's 
intimate understanding of the intricacies of IBM's Power Systems and large data sets in a free-wheeling discussion that will answer 
all your questions on this complex subject. 


> http://geekguide.linuxjournal.com/content/maximizing-nosql-clusters-large-data-sets 
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puppet Puppet Labs 2015 State of DevOps Report 


Sponsor: Puppet Labs 


DevOps represents a profound change from the way most IT departments have traditionally worked: from siloed teams and 
high-anxiety releases to everyone collaborating on uneventful and more frequent releases of higher-quality code. It doesn't 
matter how large or small an organization is, or even whether it's historically slow moving or risk averse — there are ways to 
adopt DevOps sanely, and get measurable results in just weeks. 


> http://geekguide.linuxjournal.com/content/how-build-high-performing-it-teams-including-new-data- 
it-performance-puppet-labs-2015-state 
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redislabs Comparing NoSQL Solutions 
In a Real-World Scenario 


Sponsor: RedisLabs | Topic: Web Development | Author: Avalon Consulting 


Specializing in cloud architecture, Emind Cloud Experts is an AWS Advanced Consulting Partner and a Google Cloud 
Platform Premier Partner that assists enterprises and startups in establishing secure and scalable IT operations. The 
following benchmark employed a real-world use case from an Emind customer. The Emind team was tasked with 
the following high-level requirements: 


e Support a real-time voting process during massive live events 

(e.g., televised election surveys or “America Votes” type game shows). 
e Keep voters’ data anonymous but unique. 
e Ensure scalability to support surges in requests. 


> http://geekguide.linuxjournal.com/content/comparing-nosql-solutions-real-world-scenario 


98 / FEBRUARY 2016 / WWW.LINUXJOURNAL.COM 


FREE DOWNLOADS 


NEW Forrester Study! 

==S== Linux Management with Red Hat Satellite: 
== ===. Measuring Business Impact and ROI 

Achieving Application Delivery Velocity with a 482% ROI 


IBM commissioned Forrester Consulting to conduct its Total Economic Impact™ (TEI) study that examines and quantifies potential 
return on investment (ROI) for IBM UrbanCode Deploy within an enterprise DevOps environment. The study determined that a 
composite organization, based on the customers interviewed, experienced an ROI of 482%! 


Read the Forrester Consulting study and learn learn how these enterprise organizations achieved: 


¢ 97% reduction in the cost of releases. 
e Reduction in the risk of failed deployments. 
e 75% faster deployment times. 


See how IBM UrbanCode brings deployment velocity while reducing release costs. 


> http://devops.linuxjournal.com/devops/total-economic-impacttm-ibm-urbancode 


Mobile to Mainframe DevOps for Dummies 


In today’s era of digital disruption empowered by cloud, mobile, and analytics, it's imperative for enterprise 
organizations to drive faster innovation while ensuring the stability of core business systems. While innovative 
systems of engagement demand speed, agility and experimentation, existing systems of record require similar 
attributes with additional and uncompromising requirements for governance and predictability. In this new book 
by Rosalind Radcliffe, IBM Distinguished Engineer, you will learn about: 


Mobile to 
Mainframe DevOps 


e Responding to the challenges of variable speed IT. 

e Why the mainframe is a unique and ideal platform for developing hybrid cloud applications. 

e How mobile front ends can rejuvenate back-end systems to reach new customers. 

e And, special considerations for using a DevOps approach to accelerate mainframe software delivery. 


> http://devops.linuxjournal.com/devops/mobile-mainframe-devops-dummies 


BRAND-NEW EDITION! 
DevOps For Dummies - New Edition with SAFe® 


In this NEW 2nd edition, learn why DevOps is essential for any business aspiring to be lean, agile, and capable of responding 
rapidly to changing customers and marketplace. 


Download the E-book to learn about: 


e The business need and value of DevOps. 
e¢ DevOps capabilities and adoption paths. 
e How cloud accelerates DevOps. 

e The Ten DevOps myths. 

e And more. 


> http://devops.linuxjournal.com/devops/devops-dummies-new-edition-safe 
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Giving Silos 
Their Due 


DOC SEARLS 


We’re not wrong in our principles or our work. But, we may 
have to admit that silos have proven right in some big ways. 


wo things | got way wrong, 

Ts back. 
One was Linux on the Desktop 

(LOTD, http://www.linuxjournal.com/ 
googlesearch?s=lotd). Around 
the turn of the Millennium, | 
predicted big successes for LOTD 
and Linux on the Laptop (LOTL, 
http://www. linuxjournal.com/ 
article/7464)—and continued to 
do the same, annually, until | 
gave up. Here’s how my optimism 
looked in January 2003 
(http://www.linuxjournal.com/ 
article/6548): 


Is 2003 the year we see 
big-name hardware companies 
selling Linux as aggressively as 
they sell Windows—by which 

| mean Linux PCs show up in 
stores and on the front pages 
of Web sites? 
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Paul Saffo famously said we 
overestimate in the short term and 
underestimate in the long. But I’m 
going to go out on a limb and say 
the long term is mostly behind us. 
(How many years have we been 
waiting for LOTD to take off?) 
Somebody is going to break the ice 
this year. My own instinct says it'll 
be IBM, mostly because there must 
be serious demand for LOTD (as 
well as improvised solutions with 
it) inside the company. You can’t 
have that many engineers walking 
around with Linuxified ThinkPads 
and not get around to selling them 
at some point. 


I'm betting Dell will be next. Then 
Gateway. Then HP.... 


Didn't happen. Sure, some 


of the bigs sold personal Linux 


EOF 


Which brings me to the second thing I got wrong: 
expecting the world to prefer flat, distributed, 
open and free technology ecosystems over silo'd 
centralized, closed and proprietary ones. 


hardware (and still do), but not “as 
aggressively as they sell Windows”. 
If you're not a Linux geek today, 
your choices of desktops and 
laptops are contained by two silo’d 
operating systems from just two 
companies: Apple and Microsoft. 
Hardware choices are also narrower 
since Microsoft decided to follow 
Apple into the hardware business. 
The center of device use gravity has 
also moved over to mobile, where 
most of us are using gear running 
just two operating system platforms: 
Apple’s iOS and Google’s Android. 
True, Android is based on Linux (a 
victory we gladly claim), and the range 
of devices running Android is huge 
(http://techcrunch.com/2014/08/21/ 
opensignal-2014-android- 
ecosystem-report), but Android is still 
Google's show. The network frontier 
is also mostly fenced off by mobile 
carriers and device suppliers who 
believe, almost across the board, 
that captive customers and users are 


more valuable than free ones—and 
everybody agrees, at least tacitly, 
that “free market” means “your 
choice of captor”. 

Which brings me to the second 
thing | got wrong: expecting the 
world to prefer flat, distributed, 
open and free technology ecosystems 
over silo'd centralized, closed and 
proprietary ones. One example is 
XMPP (https://en.wikipedia.org/ 
wiki/XMPP), originally called Jabber. 
It was meant to bridge or replace 
all the competing proprietary 
instant-messaging systems in use 
at that time: AOL’s AIM and ICQ, 
Microsoft's MSN, Yahoo's Messenger, 
Apple's whatever-it-was (now called 
iMessage) and so on. 

Didn't happen. Look up “Jabber” 
today at LinuxJournal.com and 
you'll get a /ot of results, also 
from back around the turn of the 
Millennium, mostly written by me. 
In those days, many of us had full 
confidence that Jabber/XMPP would 
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Wikipedia currently lists 55 sites, services, 
protocols and programs that do chat. There is no 
underlying protocol for interoperation among them. 


do for instant messaging (aka chat) 
what SMTP/POP3/IMAP did for e-mail 
and HTTP/HTML and its successors 
did for publishing and all the other 
things one can do on the World 
Wide Web. We would have a nice 
flat, distributed and universal 
standard that people could employ 
any way they wanted, including 

on their own personal hardware 

and software, with countless 
interoperable systems and no natural 
barriers to moving data easily from 
any one system to any other. 

Didn't happen. Yes, XMPP turned 
into a widely used standard and 
either supplied or inspired lots 
of messaging systems. But today, 
instant messaging is almost entirely 
silo‘d, and it’s more popular 
than ever. Facebook (including 
WhatsApp), Twitter, Google and 
LinkedIn have their own messaging 
systems inside their own “social” 
silos. So do Apple and Microsoft 
(including Skype). Tencent Holdings, 
the giant Web portal company in 
China, has Tencent QQ, WeChat and 
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Qzone (a social site with messaging 
built in). Some interoperate, but 
most don’t, since that would free 
their users from captivity. 

Wikipedia currently lists 55 sites, 
services, protocols and programs 
that do chat. There is no underlying 
protocol for interoperation among 
them. Nor do any makers seem 
especially interested. Google Talk 
(https://developers.google.com/talk) 
interoperated with other XMPP 
systems, but then it was 
replaced with Google Hangouts 
(http://windowspbx.blogspot. 
com/2013/05/hangouts-wont- 
hangout-with-other.html), 
which lacks support for XMPP. 

So, while XMPP.org says “it’s not 

the end of XMPP for Google Talk” 
(https://xmpp.org/2015/03/no-its-not- 
the-end-of-xmpp-for-google-talk), 

it hardly matters if Google Talk is 
itself dead. 

The last nail in XMPP’s coffin, 
perhaps, is this from Facebook 
(https://developers.facebook.com/ 
docs/chat): 


On April 30, 2014, we announced 
the deprecation of the XMPP 
Chat API as part of the release of 
Platform API v2.0. 


After April 30th, 2015 apps will 
no longer be able to access the 
service or API. This includes both 
access to chat.facebook.com and 
the xmpp_login permission. 


We recommend people access 
Facebook Messages on the 
desktop via Facebook.com 

or Messenger.com. 


On the other hand (or on another 
tentacle), the number and variety 
of chat platforms and apps is huge, 
and growing. Apple’s App Store 
currently lists 171 chat apps, while 
Google's Play Store for Android lists 
165 (https://play.google.com/store/ 
search?q=chat&hl=en). Nearly all 
of them are proprietary, and many 
also depend on (or work inside) large 
silo'd proprietary services. Listen for 
complaints about all this and you'll 
hear crickets. 

APIs are also silos. As more 
and more of the connected world 
comes to depend on Whatever as 
a Service, “the cloud” and other 
remote and centralized services, the 
very idea of distributed capacities 
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So now the chat ecosystem is a forest of silos 
standing on free and open-source geology, but 
not built with the same rock. 


on standalone open-source (or 
open-source-based) hardware 
and software that individuals 
or organizations fully control as 
independent and sovereign entities 
in the world, seems terribly retro, 
utopian, or both. 

Brian Behlendorf says, wisely, 
that the ideal is “minimum viable 
centralization”. | think that should 
be a transcendent design principle. 
But the centralized thing is too 
often the easiest to make or to rely 
on (as is the case with APIs and big 
silo'd platforms). So now the chat 
ecosystem is a forest of silos standing 
on free and open-source geology, but 
not built with the same rock. 

What may be hardest for 
old-skool types like me to admit is that 
silos are now the preferred method 
for inventing and promulgating 
technologies supporting uses 
that scale and invite incompatible 
competitors to do the same, while 
the marketplace as a whole doesn’t 
have a big problem with any of it. 

I'll never believe silos are the best 
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way to make the world work in the 
long run. And I'll always believe 
that the flat distributed world built 
on free and open stuff is the most 
supportive and fertile base on which 
to build the best and broadest 
range of goods and services. But I’m 
past believing that this will ever be 
obvious to the world of developers, 
businesses and governing bodies. 
Muggles have always outnumbered 
wizards and always will. 

Being out of sight and mind for 
most of the world doesn’t make us 
wrong. In fact, it might just make 
us right in ways most will never see, 
no matter how much they depend 
on what we do.@ 


Doc Searls is Senior Editor of Linux Journal. He is also a 
fellow with the Berkman Center for Internet and Society at 
Harvard University and the Center for Information Technology 
and Society at UC Santa Barbara. 
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